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EXECUTIVE  SUMMARY 


1.0  Introduction 

This  summary  serves  as  a  stand-alone  document,  as  well  as  part  of  this  report.  For  that  reason, 
the  reader  will  find  some  duplication  of  verbiage  and  figures  between  the  summary  and  the  lull 
report. 

2.0  JADS  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  deputy  director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  in  October 
1994  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for  support 
of  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation  (OT&E).  The 
program  is  Air  Force  led  with  Army  and  Navy  participation.  JADS  Joint  Test  Force  (JTF) 
manning  currently  includes  18  Air  Force  military,  4  Air  Force  civilians,  12  Army  military,  and  1 
Navy  civilian.  Science  Applications  International  Corporation  and  the  Georgia  Tech  Research 
Institute  provide  contracted  technical  support.  The  program  is  currently  scheduled  to  end  in 
March  2000. 

The  JADS  JTF  is  directly  investigating  ADS  applications  in  three  slices  of  the  test  and  evaluation 
(T&E)  spectrum:  the  System  Integration  Test  (SIT)  which  explored  ADS  support  of  air-to-air 
missile  testing;  the  End-to-End  (ETE)  Test  which  is  investigating  ADS  support  for  command, 
control,  communications,  computers,  intelligence,  surveillance,  and  reconnaissance  (C4ISR) 
testing;  and  the  Electronic  Warfare  (EW)  Test  which  is  exploring  ADS  support  for  EW  testing. 
The  JTF  is  also  chartered  to  observe  or  participate  at  a  modest  level  in  ADS  activities  sponsored 
and  conducted  by  other  agencies  in  an  effort  to  broaden  conclusions  developed  in  the  three 
dedicated  test  areas. 

Phase  3,  the  transition  phase  of  the  ETE  Test,  is  the  subject  of  this  summary  report. 

3.0  ETE  Test  Overview 

The  ETE  Test  is  designed  to  evaluate  the  utility  of  ADS  to  support  testing  of  C4ISR  systems. 
The  test  uses  the  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one 
component  of  a  representative  C4ISR  system.  The  ETE  Test  also  evaluates  the  capability  of  the 
JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a  distributed  test  of  this  type  and 
remotely  monitor  and  analyze  test  results. 

The  ETE  Test  consists  of  four  phases.  Phase  1  developed  or  modified  the  components  needed  to 
develop  the  ADS  test  environment.  Phase  2  used  the  ADS  test  environment  to  evaluate  the  utility 
of  ADS  to  support  DT&E  and  early  OT&E  of  a  C4ISR  system  in  a  laboratory  environment. 
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Phase  3  transitioned  portions  of  the  architecture  to  the  E-8C  aircraft,  ensured  that  the  components 
functioned  properly,  and  checked  that  the  synthetic  environment  interacted  properly  with  the 
aircraft  and  actual  light  ground  station  module  (LGSM).  Phase  4  will  evaluate  the  ability  to 
perform  test  and  evaluation  of  the  E-8C  and  LGSM  in  a  synthetically  enhanced  live  test 
environment. 

4.0  Overview  of  ETE  Test  Phase  3 
4.1  Purpose 

Phase  3  provided  an  iterative  step  in  determining  the  utility  of  ADS  to  the  T&E  of  a  C4ISR 
system.  During  this  phase,  portions  of  the  architecture  were  transferred  to  the  E-8C  aircraft,  the 
components  were  checked  to  make  sure  they  functioned  properly,  and  the  synthetic  environment 
was  checked  to  make  sure  it  interacted  properly  with  the  aircraft  and  the  LGSM. 

JADS  Issue  1.  What  is  the  present  utility  of  ADS,  including  DIS,  for  T&E? 

JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including  DIS, 
during  test  execution. 

JADS  Objective  1-2.  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E.  (This 
objective  was  not  applicable  to  Phase  3  of  the  ETE  Test.) 

JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS 
for  T&E? 

JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS  performance  for 
T&E.  This  objective  was  broken  down  into  subobjectives. 

JADS  Subobjective  2-1-1.  Assess  player  instrumentation  and  interface  performance 
constraints  and  concerns.  (This  subobjective  was  not  applicable  to  Phase  3  of  the  ETE 
Test.) 

JADS  Subobjective  2-1-2.  Assess  network  and  communications  performance  constraints 
and  concerns. 

JADS  Subobjective  2-1-3.  Assess  the  impact  of  ADS  reliability,  availability  and 
maintainability  on  T&E. 

JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support  systems 
for  T&E.  This  objective  was  broken  down  into  subobjectives. 

JADS  Subobjective  2-2-1.  Assess  the  critical  constraints  and  concerns  regarding  ADS 
data  management  and  analysis  systems. 
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JADS  Subobjective  2-2-2.  Assess  the  critical  constraints  and  concerns  regarding 
configuration  management  of  ADS  test  assets.  (This  subobjective  was  not  applicable  to 
Phase  3  of  the  ETE  Test.) 

JADS  Objective  2-3.  Develop  and  assess  methodologies  associated  with  ADS  for  T&E. 
(This  objective  was  not  applicable  to  Phase  3  of  the  ETE  Test.) 

4.2  Approach 

Figure  ES-1  provides  an  overview  of  the  Phase  3  ETE  Test  synthetic  environment. 


AFATDS  =  Advanced  Field  Artillery  Tactical  Data  System 
Bn  =  battalion 

SCDL  =  surveillance  control  data  link 

TAC  =  target  analysis  cell 

WSMR  =  White  Sands  Missile  Range,  New  Mexico 


ATACMS  =  Army  Tactical  Missile  System 
Janus  =  interactive,  computer-based  simulation  of  combat  operations 
T-l  -  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1 .544 
megabits  per  second 

TRAC  -  U.S  Army  Training  and  Doctrine  Command  Analysis  Center 


Figure  ES-1.  ETE  Test  Phase  3  Synthetic  Environment 


The  Test  Control  and  Analysis  Center  in  Albuquerque,  New  Mexico,  provided  test  control. 


The  Joint  STARS  E-8C  simulation,  called  the  Virtual  Surveillance  Target  Attack  Radar  System 
(VSTARS),  represented  the  radar  subsystem  of  the  Joint  STARS  E-8C  in  a  laboratory 
environment.  It  was  composed  of  a  distributed  interactive  simulation  network  interface  unit 
(NIU),  a  radar  processor  simulator  and  integrator  (RPSI)  that  contained  the  two  real-time  radar 
simulations  with  necessary  databases,  and  various  simulations  of  E-8C  processes.  Figure  ES-2 
provides  more  information  on  the  VSTARS  architecture. 
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CDP  -  central  data  processor 
ESPDU  =  entity  state  protocol  data  unit 
MTI  =  moving  target  indicator 
SCDL  =  surveillance  control  data  link 
SMO  -  system  management  officer 


DIS  =  distributed  interactive  simulation 
M&IS  -  management  and  integration  software 
SAR  =  synthetic  aperture  radar 
SM&C  -  system  management  and  control 

T-l  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits  per  second 


Figure  ES-2.  VSTARS  Architecture 

The  approach  taken  during  the  ETE  Test  Phase  3  was  to  migrate  certain  software  components  of 
VSTARS,  specifically  the  air  network  interface  unit  (ANIU)  and  the  radar  processor  simulator 
and  integrator  (RPSI),  from  the  laboratory  Alpha  workstations  to  the  primary  mission  equipment 
on  the  T3  E-8C  aircraft.  In  addition,  the  ground  network  interface  unit  (GNIU)  software  would 
be  separated  from  VSTARS  and  migrated  to  an  Alpha  workstation  collocated  with  a  satellite 
transceiver. 

Once  the  migration  was  completed,  each  component  was  tested  in  isolation  and  then  tested  as  a 
part  of  the  complete  environment.  Specifically,  the  network  to  GNIU  link  was  tested,  and  it  was 
verified  that  the  GNIU  was  issuing  a  VSTARS  data  packet  for  each  protocol  data  unit  (PDU) 
received.  The  GNIU  to  satellite  transceiver  to  satellite  transceiver  to  ANIU  was  also  tested,  and 
it  was  verified  that  VSTARS  data  packets  were  received  and  processed  by  the  ANIU.  Finally,  the 
RPSI  was  tested,  and  it  was  verified  that  it  processed  the  data  and  generated  the  appropriate  radar 
reports.  Once  all  components  were  shown  to  be  working,  the  entire  environment  was  tested  using 
PDUs  generated  at  the  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center  (TRAC), 
White  Sands  Missile  Range  (WSMR),  sent  to  Northrop  Grumman  and  then  via  satellite  to  the 
aircraft. 

The  last  part  of  Phase  3  consisted  of  a  series  of  system  integration  tests  (SIT)  conducted  by  the 
Joint  STARS  Joint  Test  Force  and  a  complete  verification  and  validation  (V&V)  of  the  RPSI. 
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The  SITs,  conducted  using  the  T-3  aircraft  and  a  medium  ground  station  module  (MGSM),  were 
to  determine  if  the  software  changes  and  additions  made  to  the  radar  build  in  any  way  affected  the 
performance  of  the  radar  and  operator  workstations.  The  Y&V  was  to  ensure  that  the  ADS- 
enhanced  radar  system  met  the  requirements  and  acceptability  criteria  established  by  the  ETE  Test 
team. 

Fire  support,  provided  by  the  Advanced  Field  Artillery  Tactical  Data  System  (AFATDS),  and  a 
LGSM  were  stationed  at  Fort  Hood,  Texas. 

Communications  among  these  command,  control,  communications,  computers  and  intelligence 
(C4I)  systems  employed  such  doctrinally  correct  means  as  the  CGS-100,  a  subsystem  of  the 
Compartmented  All  Source  Analysis  System  Message  Processing  System  (CAMPS),  remote 
workstations,  and  AFATDS  message  traffic.  The  AFATDS  messages  were  transmitted  between 
the  AFATDS  located  at  Fort  Hood  and  the  AFATDS  located  at  Fort  Sill  using  actual  tactical 
protocols  rather  than  distributed  interactive  simulation  (DIS)  PDUs.  Also,  the  surveillance 
control  data  link  (SCDL)  messages  were  transmitted  between  VSTARS  and  the  LGSM  using  a 
dedicated  link,  a  special-purpose  interface,  and  the  actual  tactical  protocols. 

The  Tactical  Army  Fire  Support  Model  (TAFSM)  simulation  at  Fort  Sill  modeled  the  Army 
Tactical  Missile  System  (ATACMS)  battalion  (Bn)  and  sent  the  fire  and  detonate  PDUs  to  the 
Janus  6.88D  simulation.  In  turn,  Janus  modeled  the  engagement  results  and  reflected  the  results 
in  the  synthetic  environment. 

5.0  ETE  Test  Phase  3  Results 

5.1  Schedule 

The  overall  ETE  Test  schedule  is  presented  in  Figure  ES-3.  Phase  3  testing  proceeded  as 
scheduled. 
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Figure  ES-3.  ETE  Test  Schedule 


5.2  Fulfillment  of  Test  Objectives 

All  ETE  Test  Phase  3  objectives  were  met.  Phase  3  provided  an  iterative  step  in  determining  the 
utility  of  ADS  to  the  T&E  of  a  C4I  system.  This  phase  ensured  that  the  RPSI  functioned  properly 
onboard  the  E-8C,  and  that  the  synthetic  environment  interacted  correctly  with  the  aircraft. 

6.0  Lessons  Learned 

6.1  Technical 

Testers  should  carefully  plan  the  development  of  the  simulations  and  links  comprising  their  ADS 
environment.  During  test  execution,  they  must  ensure  that  the  time  sources  are  synchronized  and 
continuously  monitor  PDU  traffic.  The  distributed  nature  of  ADS  testing  necessitates  special 
equipment  for  network  check-out  and  verification  and  requires  strict  configuration  control  of 
analysis  tools  and  collected  data. 


6 


6.2  Infrastructure  and  Process 


ADS  test  planning  should  be  detailed  enough  to  encompass  key  requirements  at  the  earliest 
possible  stages,  yet  flexible  enough  to  accommodate  unexpected  situations  during  test  execution. 

A  conservative  development  approach  is  recommended  —  accomplish  risk  reduction  activities 
before  each  ADS  test  and  let  each  ADS  test  build  on  the  success  of  earlier  experiments. 
Successful  test  execution  requires  effective  intemode  communication,  test  and  resource  control, 
and  data  management  procedures. 

7.0  Conclusions 

7.1  Utility 

A  review  of  ETE  testing  to  date  indicates  that  an  ADS  environment  can  enhance  C4ISR  system 
DT&E  and  OT&E.  In  comparison  with  conventional  tests,  ADS  allows  testers  to  examine  C4ISR 
systems  under  realistic  conditions  for  longer  periods  of  time,  over  far  larger  battlespaces,  and  at  a 
much  lower  cost.  This  versatile  technology  can  provide  test  environments  that  include  large 
numbers  of  entities,  entities  operating  under  realistic  but  unsafe  conditions,  and  joint  and 
combined  operations.  ADS  provides  C4ISR  system  testers  with  greater  flexibility  in  designing, 
executing,  and  analyzing  their  tests.  During  DT&E,  ADS  allows  for  more  realistic  compliance 
testing  of  C4ISR  subsystems  and  efficient  implementation  of  the  test-fix-verify  cycle  for  software 
development. 

7.2  Technical 

The  ETE  Test  network  was  highly  reliable  during  Phase  3  testing. 

As  expected,  the  Phase  3  testing  at  the  Northrop  Grumman  node  showed  that  all  of  the  available 
satellite  link  bandwidth  was  required  for  data  transmission,  and  that  buffering  was  needed  at  times 
to  handle  periods  of  heavy  scenario  activity.  Without  buffering,  the  satellite  link  exhibited  a 
normal  latency  of  around  two  seconds.  With  buffering,  the  latency  approached  six  seconds. 
Neither  of  these  latencies  was  observable  in  the  radar  reports,  indicating  that  the  ETE  Test 
synthetic  environment  is  very  tolerant  of  latencies  in  this  range.  However,  ADS  test  planners 
need  to  consider  these  factors  when  testing  other  C4ISR  systems  involving  satellite  links. 

7.3  Infrastructure 

Based  on  cumulative  ETE  test  experience,  ADS  testing  reduces  the  need  for  large  numbers  of 
fielded  personnel  and  vehicles,  when  compared  with  conventional  testing.  The  ability  to 
automatically  collect  and  analyze  test  data  also  reduces  the  number  of  people  required  for  setup, 
execution,  and  analysis.  ADS  test  success  relies  on  well-organized  test  control  and  data 
management  procedures.  Distributed  testing  requires  sophisticated  instrumentation,  trained 
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personnel  to  operate  and  maintain  that  equipment,  and  funds  to  support  personnel  and  equipment 
at  distant  test  nodes. 
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1.0  Introduction 


1.1  Joint  Advanced  Distributed  Simulation  Overview 

The  Joint  Advanced  Distributed  Simulation  (JADS)  Joint  Test  and  Evaluation  (JT&E)  was 
chartered  by  the  deputy  director,  Test,  Systems  Engineering  and  Evaluation  (Test  and 
Evaluation),  Office  of  the  Under  Secretary  of  Defense  (OSD)  (Acquisition  and  Technology)  in 
October  1994  to  investigate  the  utility  of  advanced  distributed  simulation  (ADS)  technologies  for 
support  of  developmental  test  and  evaluation  (DT&E)  and  operational  test  and  evaluation 
(OT&E).  The  program  is  Air  Force  led  with  Army  and  Navy  participation.  JADS  Joint  Test 
Force  (JTF)  manning  currently  includes  18  Air  Force  military,  4  Air  Force  civilians,  12  Army 
military,  and  1  Navy  civilian.  Science  Applications  International  Corporation  (SAIC)  and  the 
Georgia  Tech  Research  Institute  provide  contracted  technical  support.  The  program  is  currently 
scheduled  to  end  in  March  2000. 

The  JADS  JT&E  charter  focuses  on  three  issues:  what  is  the  present  utility  of  ADS,  including 
distributed  interactive  simulation  (DIS),  for  test  and  evaluation  (T&E);  what  are  the  critical 
constraints,  concerns,  and  methodologies  when  using  ADS  for  T&E;  and  what  are  the 
requirements  that  must  be  introduced  into  ADS  systems  if  they  are  to  support  a  more  complete 
T&E  capability  in  the  future.  From  these,  issues,  objectives  and  measures  have  been  developed  to 
guide  the  evaluation. 

The  JADS  JTF  is  directly  investigating  ADS  applications  in  three  slices  of  the  T&E  spectrum:  the 
System  Integration  Test  (SIT)  which  explored  ADS  support  of  air-to-air  missile  testing;  the  End- 
to-End  (ETE)  Test  which  investigated  ADS  support  for  command,  control,  communications, 
computers,  and  intelligence,  surveillance  and  reconnaissance  (C4ISR)  testing;  and  the  Electronic 
Warfare  (EW)  Test  which  is  exploring  ADS  support  for  EW  testing.  Each  test  applied  the  JADS 
objectives  and  measures  as  appropriate  to  conduct  its  evaluation.  The  JTF  is  also  chartered  to 
observe  or  participate  at  a  modest  level  in  ADS  activities  sponsored  and  conducted  by  other 
agencies  in  an  effort  to  broaden  conclusions  developed  in  the  three  dedicated  test  areas. 

The  JADS  ETE  Test  is  the  subject  of  this  report  and  is  described  in  the  next  section;  the  following 
is  a  brief  synopsis  of  the  SIT  and  EW  Test. 

The  SIT  evaluated  the  utility  of  using  ADS  to  support  cost-effective  testing  of  an  integrated 
missile  weapon/launch  aircraft  system  in  an  operationally  realistic  scenario.  The  SIT  also 
evaluated  the  capability  of  the  JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a 
distributed  test  of  this  type  and  to  remotely  monitor  and  analyze  test  results.  The  SIT  consisted 
of  two  phases  each  of  which  culminated  in  three  flight  missions.  The  missions  simulated  a  single 
shooter  aircraft  launching  an  air-to-air  missile  against  a  single  target  aircraft.  In  the  Linked 
Simulators  Phase  (LSP),  the  shooter,  target,  and  missile  were  all  represented  by  simulators.  In 
the  Live  Fly  Phase  (LFP),  the  shooter  and  target  were  represented  by  live  aircraft  and  the  missile 
by  a  simulator. 
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The  EW  Test  is  evaluating  the  utility  of  ADS  in  a  distributed  EW  environment.  The  first  phase 
was  open  air  testing  to  develop  a  performance  baseline  for  two  subsequent  test  phases.  The  first 
distributed  test  phase  employed  a  linked  architecture  using  Department  of  Defense’s  (DoD)  high 
level  architecture  (HLA)  which  included  a  digital  simulation  model  of  the  ALQ-131  self¬ 
protection  jammer,  threat  simulation  facilities,  and  constructive  models  which  supported 
replication  of  the  open  air  environment.  In  the  second  phase,  an  installed  systems  test  facility  was 
substituted  for  the  digital  model.  In  both  distributed  test  architectures,  system  performance  data 
were  compared  with  live  fly  data  for  verification  and  validation  (V&V). 

1.2  Test  Overview 

The  ETE  Test  is  designed  to  evaluate  the  utility  of  ADS  to  support  testing  of  C4ISR  systems.  It 
will  conduct  its  T&E  utility  evaluation  in  an  ADS-enhanced  test  environment,  using  the  Joint 
Surveillance  Target  Attack  Radar  System  (Joint  STARS)  as  one  component  of  a  representative 
C4ISR  environment.  The  ETE  Test  will  also  evaluate  the  capability  of  the  JADS  TCAC  to 
control  a  distributed  test  of  this  type  and  to  remotely  monitor  and  analyze  test  results. 

The  ETE  Test  is  using  distributed  simulation  to  assemble  an  enhanced  environment  for  testing 
C4ISR  systems.  The  intent  is  to  provide  a  complete,  robust  set  of  interfaces  from  sensor  to 
weapon  system,  including  the  additional  intermediate  nodes  that  would  be  found  in  a  tactical 
engagement.  The  test  will  trace  a  thread  of  the  complete  battlefield  process  from  target  detection 
to  target  assignment  and  engagement  at  corps  level  using  ADS.  It  will  allow  the  tester  to  evaluate 
the  thread  as  a  whole  or  the  contribution  of  any  of  the  parts  individually  and  to  evaluate  what 
effects  an  operationally  realistic  environment  has  on  the  system  under  test. 

The  ETE  Test  is  designed  to  add  additional  entities  in  a  seamless  manner  to  the  battlefield  seen  by 
Joint  STARS.  In  addition,  adding  some  of  the  complementary  suite  of  other  command,  control, 
communications,  computers  and  intelligence  (C4I)  and  weapon  systems  with  which  Joint  STARS 
would  interact  will  enable  the  test  team  to  evaluate  the  utility  of  an  ADS-enhanced  test 
environment. 

The  test  concept  (Figure  1)  used  ADS  to  supplement  the  operational  environment  experienced  by 
the  E-8C  and  light  ground  station  module  (LGSM)  operators.  By  mixing  available  five  targets 
with  targets  generated  by  a  constructive  model,  a  battle  array  approximating  the  major  systems 
present  in  a  notional  corps  area  of  interest  can  be  presented.  By  constructing  a  network  with 
nodes  representing  appropriate  C4I  and  weapon  systems,  a  more  robust  cross  section  of  players  is 
available  for  interaction  with  the  E-8C  and  LGSM  operators. 
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E-8C 


Test  Control 
&  Analysis 


ATACMS  =  Army  Tactical  Missile  System  FDC  =  fire  direction  center  SCDL  =  surveillance  control  data  link 

TAC  =  target  analysis  cell 


Figure  1.  ETE  Test  Conceptual  Model 

Several  components  were  required  to  create  the  ADS-enhanced  operational  environment  used  in 
the  ETE  Test.  In  addition  to  Joint  STARS,  the  ETE  Test  required  a  validated  simulation  capable 
of  generating  entities  representing  the  rear  elements  of  a  threat  force.  As  discussed  in  Section 
1.3.1,  the  ETE  Test  team  selected  the  Janus  simulation  for  this  requirement.  Also,  simulations  of 
the  Joint  STARS  moving  target  indicator  (MTI)  radar  and  synthetic  aperture  radar  (SAR)  were 
needed  to  insert  the  simulated  entities  into  the  radar  stream  onboard  the  E-8C  while  it  was  flying  a 
live  mission.  Other  capabilities  used  to  support  the  test  include  simulations  or  subsets  of  the 
Army’s  artillery  command  and  control  process  and  a  simulation  of  the  Army  Tactical  Missile 
System  (ATACMS).  Communications  among  these  simulations  are  accomplished  using  such 
doctrinally  correct  means  as  the  CGS-100,  a  subsystem  of  the  Compartmented  All  Source 
Analysis  System  (ASAS)  Message  Processing  System  (CAMPS),  remote  workstations  (RWSs), 
and  Advanced  Field  Artillery  Tactical  Data  System  (AFATDS)  message  traffic. 

The  ETE  Test  consists  of  four  phases.  Phase  1  developed  or  modified  the  components  that 
allowed  the  mix  of  live  and  simulated  targets  at  an  E-8C  operator’s  console  and  an  LGSM 
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operator’s  console.  Phase  2  evaluated  the  utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a 
C4ISR  system  in  a  laboratory  environment.  Phase  3  transitioned  portions  of  the  architecture  to 
the  E-8C  aircraft,  ensured  that  the  components  functioned  properly,  and  checked  that  the 
synthetic  environment  properly  interacted  with  the  aircraft  and  the  actual  LGSM.  Phase  4  will 
evaluate  the  ability  to  perform  test  and  evaluation  of  the  E-8C  and  LGSM  in  a  synthetically 
enhanced  operational  environment  using  typical  operators. 

1.3  Phase  1  Overview 

During  Phase  1,  software  and  hardware  needed  to  establish  the  ETE  Test  ADS  environment  were 
developed,  modified,  and  integrated.  In  addition,  Phases  2  through  4  were  planned. 

The  ETE  Test  ADS  environment  components  developed  during  Phase  1  included  a  constructive 
simulation  to  provide  virtual  targets,  an  E-8C  simulation  called  the  Virtual  Surveillance  Target 
Attack  Radar  System  (VSTARS),  an  interface  to  allow  surveillance  control  data  link  (SCDL) 
traffic  to  be  exchanged  between  VSTARS  and  the  ground  station  model  (GSM),  and  an  ADS 
network  suitable  for  integration  and  testing. 

More  detailed  information  on  Phase  1  can  be  found  in  the  End-to-End  Interim  Report,  Phase  1, 
August  1998,  available  at  http://www.JADS.abq.com.  (After  1  March  2000  refer  requests  to 
Headquarters  Air  Force  Operational  Test  and  Evaluation  Center  (HQ  AFOTEC)/HO,  8500 
Gibson  Blvd  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558,  or  SAIC  Technical  Library, 
2001  North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

1.3.1  Phase  1  Approach 

The  JADS  ETE  Test  team  developed  requirements  for  a  constructive  simulation  and  then 
evaluated  available  simulations  against  these  requirements.  The  Janus  simulation,  developed  and 
managed  by  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 
(TRAC),  White  Sands  Missile  Range  (WSMR),  New  Mexico,  was  selected  as  the  simulation  best 
able  to  be  modified  to  meet  JADS’  requirements.  TRAC-WSMR  expanded  the  Janus  scenario 
driver  into  Janus  6.88D,  a  constructive  simulation  capable  of  supporting  up  to  10,000  individual 
entities  with  a  distributed  interactive  simulation  (DIS)  interface  to  the  ETE  Test  environment. 

The  JADS  ETE  Test  team  investigated  existing  simulations  of  Joint  STARS  and  determined  that 
none  of  them  met  the  needed  fidelity  requirements.  Based  upon  a  JADS  ETE  Test  concept, 
Northrop  Grumman,  the  developer  of  the  E-8C,  created  a  laboratory  emulation  of  the  E-8C  radar 
subsystem  called  the  radar  simulation  processor  and  integrator  (RPSI).  VSTARS  is  a  laboratory 
emulation  of  the  E-8C  aircraft  that  contains  the  RPSI  and  other  aircraft  components.  VSTARS 
can  receive  entity  state  protocol  data  units  (ESPDUs)  from  a  DIS  network  and  create  virtual 
radar  reports  that  are  displayed  on  the  Advanced  Technology  Work  Station  (ATWS)  or  an 
LGSM.  The  RPSI  and  the  air  network  interface  unit  (ANIU)  are  the  parts  of  VSTARS  that  are 
installed  on  the  aircraft.  The  RPSI  receives  radar  service  requests  (RSRs)  from  either  an  operator 
workstation  (OWS)  or  a  GSM  and  provides  radar  reports  to  the  OWS  and  GSM. 
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Phase  1  also  included  the  development  of  a  near  real-time  emulation  of  the  E-8C  synthetic 
aperture  radar  (SAR).  The  JADS  ETE  Test  team,  through  the  Advanced  Research  Projects 
Agency  (ARP A)  War  Breaker  project,  conducted  a  trade  study  of  various  existing  simulations. 
The  XPATCHES  simulation,  developed  by  Wright  Laboratories  and  Loral  Defense  Systems 
(Goodyear,  Arizona),  was  selected  as  the  best  starting  point  for  the  E-8C  SAR  emulation. 
Lockheed  Martin  Tactical  Defense  Systems  (previously  Loral  Defense  Systems),  Goodyear, 
Arizona,  developed  the  SAR  emulation  that  represents  the  Joint  STARS  SAR.  This  emulation  is 
referred  to  as  the  Advanced  Radar  Imaging  Emulation  System  (ARIES)  and  is  integrated  into  the 
RPSI. 

The  normal  means  of  exchanging  data  among  the  E-8C  and  its  associated  LGSMs  is  through  a 
line-of-sight  data  link  that  is  called  the  surveillance  control  data  link  (SCDL).  Internal  to  both  the 
E-8C  and  the  GSM,  the  data  are  handled  as  standard  Ethernet  packets  and  converted  to  SCDL 
format  prior  to  transmission.  They  are  also  bridged  over  to  a  1553  databus  prior  to  being  sent  to 
the  air  data  terminal  (ADT)  or  the  ground  data  terminal  (GDT).  Since  the  SCDL  formatted  data 
packets,  prior  to  bridging  over  to  the  1553  databus,  can  be  sent  via  a  T-l  line  from  point  to  point, 
it  was  determined  that  the  data  packets  could  be  sent  directly  from  VSTARS  to  the  GSM’s 
location.  Once  at  the  GSM’s  location,  the  data  would  be  bridged  over  to  the  1553  databus  and 
input  into  the  GSM  as  if  they  had  been  received  by  the  GDT.  Conversely,  data  originating  at  the 
GSM  would  leave  the  GSM  on  the  1553  databus  and  be  bridged  over  to  a  protocol  that  could  be 
sent  via  the  T-l  line  to  VSTARS.  Motorola  developed  the  bridge  between  the  LGSM  and 
VSTARS.  This  interface  unit  links  the  T-l  with  the  internal  1553  databus  of  the  LGSM  and 
simulates  some  of  the  functions  of  the  ground  data  terminal.  This  requires  the  LGSM  operator  to 
perform  many  of  the  normal  linking  process  prior  to  receiving  message  traffic  from  VSTARS. 

The  Phase  1  network  initially  connected  TRAC-WSMR  with  the  JADS  TCAC  in  Albuquerque, 
New  Mexico,  and  was  then  extended  from  the  TCAC  to  the  Northrop  Grumman  laboratory 
facilities  in  Melbourne,  Florida.  Late  in  Phase  1,  this  network  grew  to  include  links  from  the 
TCAC  to  Fort  Hood,  Texas,  and  Fort  Sill,  Oklahoma,  and  a  link  between  Northrop  Grumman  and 
Fort  Hood. 

1.3.2  Phase  1  Results 

Phase  1  identified  constraints  associated  with  ADS  testing.  One  key  constraint  was  the  ability  of 
the  DoD  infrastructure  to  support  ADS  test  and  evaluation.  A  measure  of  this  constraint  is  found 
in  the  amount  of  development  required  to  establish  a  synthetic  environment  with  which  to  conduct 
testing.  Phase  1  provided  insight  onto  the  development  required  to  support  a  test  of  this  type. 
Phase  1  also  demonstrated  the  application  of  a  systems  engineering  methodology  to  identify  the 
requirements  for  ADS  components,  evaluated  the  availability  of  ADS  components,  and  modified 
or  developed  the  components  to  meet  the  requirements. 

During  Phase  1  extensive  testing  was  conducted  to  establish  and  verify  the  network  configuration. 
Data  management  and  analysis  methods  were  also  examined  and  the  methods  that  were  used 
during  the  subsequent  phases  of  the  test  were  developed. 
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1.4  Phase  2  Overview 


Phase  2  of  the  ETE  Test  determined  the  utility  of  ADS  to  support  DT&E  and  early  OT&E  of  a 
C4ISR  system  in  a  laboratory-based  environment. 

More  detailed  information  on  Phase  2  can  be  found  in  the  End-to-End  Interim  Report,  Phase  2, 
February  1999,  available  at  http://www.JADS.abq.com.  (After  1  March  2000  refer  requests  to 
HQ  AFOTEC/HO,  8500  Gibson  Blvd  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558,  or 
SAIC  Technical  Library,  2001  North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

1.4.1  Phase  2  Approach 

Several  components  were  required  to  create  the  ADS-enhanced  environment  used  in  Phase  2. 
Figure  2  provides  an  overview  of  the  Phase  2  synthetic  environment. 

The  ETE  Test  used  the  Janus  6.88D  simulation  to  generate  the  entities  representing  the  elements 
in  the  rear  of  a  threat  force.  Janus  generated  ESPDUs  for  the  threat  force  which  were  transmitted 
to  the  E-8C  simulation  via  the  Test  Control  and  Analysis  Center  (TCAC).  TRAC-WSMR 
provided  the  Janus  scenario  feed. 

The  TCAC  in  Albuquerque,  New  Mexico,  provided  test  control.  The  JADS  Network  and 
Engineering  (N&E)  team  monitored  the  health  of  the  ETE  Test  network  and  ensured  that 
adequate  data  flowed  in  support  of  the  test. 


Bn  =  battalion  T-l  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1 .544  megabits  per  second 

TAC  =  target  analysis  cell 


Figure  2.  ETE  Test  Phase  2  Synthetic  Environment 

The  Joint  STARS  E-8C  simulation,  VSTARS,  represents  the  Joint  STARS  E-8C  in  a  laboratory 
environment.  It  is  composed  of  a  distributed  interactive  simulation  network  interface  unit  (NIU), 
an  RPSI  that  contains  the  two  real-time  radar  emulations  with  necessary  databases,  and  various 
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simulations  of  E-8C  processes.  Figure  3  provides  more  information  on  the  VSTARS  architecture. 
VSTARS  was  operated  at  the  Northrop  Grumman  Surveillance  and  Battle  Management  Systems 
facility  in  Melbourne,  Florida. 

Fire  support,  provided  by  the  AFATDS,  and  an  LGSM  were  stationed  at  Fort  Hood,  Texas. 


Communications  among  these  C4I  systems  employed  such  doctrinally  correct  means  as  the  CGS- 
100,  a  subsystem  of  the  CAMPS,  remote  workstations,  and  AFATDS  message  traffic.  The 
AFATDS  messages  were  transmitted  between  the  AFATDS  located  at  Fort  Hood  and  the 
AFATDS  located  at  Fort  Sill  using  actual  tactical  protocols  rather  than  DIS  PDUs.  Also,  the 
SCDL  messages  were  transmitted  between  VSTARS  and  the  LGSM  using  a  dedicated  link,  a 
special-purpose  interface,  and  the  actual  tactical  protocols. 


SMO 


DIS 

Networ^ 

ESPDUs 


CDP  -  central  data  processor  SM&C  -  system  management  and  control 

M&IS  -  management  and  integration  software  SMO  -  system  management  officer 

T-l  =  digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits  per  second 


Figure  3.  VSTARS  Architecture 

The  Tactical  Army  Fire  Support  Model  (TAFSM)  simulation  at  Fort  Sill  modeled  the  ATACMS 
battalion  and  sent  the  fire  and  detonate  PDUs  to  the  Janus  6.88D  simulation.  In  turn,  Janus 
modeled  the  engagement  results  and  reflected  the  results  in  the  synthetic  environment. 

1.4.2  Phase  2  Results 
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All  ETE  Test  Phase  2  objectives  were  met.  The  ETE  Test  ADS-enhanced  environment  was 
developed  and  tested.  An  extensive  verification  and  validation  was  conducted  of  both  the  nodes 
and  the  overall  environment,  followed  by  accreditation  of  the  environment  for  testing. 

The  ETE  Test  team  determined  that  ADS  testing  can  be  beneficial  for  test  planning,  rehearsal,  and 
execution,  and  can  result  in  valid  data  being  collected.  During  Phase  2,  they  also  identified  critical 
constraints,  concerns,  and  methodologies  associated  with  using  ADS  for  test  and  evaluation. 
Finally,  the  ETE  Test  team  utilized  and  assessed  test  control  and  data  collection  methodologies 
useful  for  ADS  testing. 
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2.0  Phase  3  Overview 


2.1  Phase  3  Purpose 

Phase  3  provided  an  iterative  step  in  determining  the  utility  of  ADS  to  the  T&E  of  a  C4I  system. 
This  phase  ensured  that  the  RPSI  functioned  properly  onboard  the  E-8C,  and  that  the  synthetic 
environment  interacted  correctly  with  the  aircraft. 

2.2  Phase  3  Approach 

Figure  4  shows  the  organizational  structure  for  reporting  and  coordination  during  Phase  3  of  the 
ETE  Test. 


Bde  =  brigade  Bn  =  battalion  Co  =  company 

D&SA  =  Depth  and  Simultaneous  Attack  DDSA  =  deputy  director,  System  Assessment  ID  =  infantry  division 

JPO  =  joint  program  office  MI  =  Military  Intelligence  PM  =  program  manager 

TEXCOM  =  U.S.  Army  Test  and  Experimentation  Command 

Figure  4.  ETE  Test  Organizational  Structure 

During  the  ETE  Test,  the  roles  and  responsibilities  of  these  organizations  are  as  follows. 


DDSA 


The  deputy  director,  System  Assessment  (DDSA)  in  Washington,  District  of  Columbia: 


•  Oversees  the  JADS  Joint  Test  and  Evaluation  (JT&E) 

•  Approves  JADS  financial  requirements 

•  Approves  the  program  test  plan  (PTP) 
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•  Oversees  the  analysis  and  reporting  of  test  results 

JADS  JTF 

The  JADS  JTF  in  Albuquerque,  New  Mexico: 

•  Conducts  overall  planning,  execution,  analysis,  and  reporting  of  the  test 

•  Manages  funding  to  accomplish  the  test 

•  Develops  and  evaluates  JADS  issues,  objectives,  measures,  and  related  data  elements 

•  Develops  and  integrates  the  components  of  the  ETE  Test  ADS  environment 

•  Establishes  necessary  communication  links  with  test  participants 

•  Operates  the  Test  Control  and  Analysis  Center  during  tests 

•  Works  with  other  organizations  in  analyzing  test  data 

•  Reports  interim  and  final  results  to  OSD 

TRAC-WSMR 

TRAC-WSMR,  New  Mexico: 

•  Develops,  tests,  and  documents  Janus  6.88D  (an  expanded  variant  of  Janus)  for  JADS 

•  Assists  in  integrating  Janus  6.88D  into  the  ETE  Test  ADS  environment 

•  Assists  in  database  conversions 

•  Assists  in  developing  vignettes 

•  Assists  in  verification,  validation,  and  accreditation  (W&A)  activities 

•  Assists  in  ETE  Test  execution 

TEXCOM  Lab 

The  U.S.  Army  Test  and  Experimentation  Command  (TEXCOM)  Lab  at  Fort  Hood,  Texas: 

•  Assists  in  scenario  and  vignette  development 

•  Assists  in  ETE  Test  execution 

D&SA  Battle  Lab 

The  Depth  and  Simultaneous  Attack  (D&SA)  Battle  Lab  at  Fort  Sill,  Oklahoma: 

•  Provides  and  operates  the  TAFSM  and  AFATDS 

•  Assists  in  the  integration  of  the  ETE  Test  ADS  environment 

•  Assists  in  W&A  activities  and  ETE  Test  execution 


U.S.  Army  III  Corps 

III  Corps  Headquarters  at  Fort  Hood,  Texas: 

•  B  Company  (Co),  303d  Military  Intelligence  (MI)  Battalion  (Bn),  504  MI  Brigade  (Bde) 
supports  the  conduct  of  ETE  Test  events  with  LGSM(s)  and  a  target  analysis  cell  (TAC)  and 
assists  in  the  integration  of  the  ETE  Test  ADS  environment 

•  504  MI  Bde  provides  a  test  environment  for  the  ETE  Test 

•  Fire  Support  4th  Infantry  Division  (ID)  provides  an  AFATDS  and  personnel  to  support  the 
ETE  Test 

Joint  STARS  Joint  Program  Office  (JPO) 

Joint  STARS  JPO,  Hanscom  Air  Force  Base  (AFB),  Massachusetts,  provides  access  to  the  Joint 
STARS  JTF  and  Northrop  Grumman. 

The  Joint  STARS  JTF  of  the  Joint  STARS  JPO  in  Melbourne,  Florida: 

•  Supports  conduct  of  testing  in  all  phases 

•  Analyzes  Joint  STARS  test  results  and  provides  evaluations  according  to  JADS  objectives 

•  Assists  in  W&A  activities 

Northrop  Grumman  Aerospace  Corporation 

Northrop  Grumman,  Electronics  and  Systems  Integration  Division  in  Melbourne,  Florida: 

•  Designed,  developed  and  integrated  the  RPSI 

•  Developed  the  VSTARS 

•  Conducts  and  assists  in  verification  and  validation  activities 

•  Assists  in  E-8C  mission  planning 

•  Operates  VSTARS  during  ETE  Test  phases 

Contracting  with  Northrop  Grumman  is  conducted  through  Rome  Laboratory  in  New  York. 

2.3  Test  Objectives 

The  JADS  issues,  test  objectives,  and  subobjectives  for  Phase  3  are  described  below.  Each 
subobjective  in  turn  encompassed  one  or  more  test  measures.  In  Section  4  these  issues, 
objectives,  subobjectives,  and  test  measures  are  discussed  in  terms  of  their  intent,  the  associated 
data  collection  methodology,  and  operational  test  results. 

JADS  Issue  1.  What  is  the  present  utility  of  ADS,  including  DIS,  for  T&E? 
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JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including  DIS, 
during  test  execution. 

JADS  Objective  1-2.  Assess  the  benefits  of  using  ADS,  including  DIS,  in  T&E.  (This 
objective  was  not  applicable  to  Phase  3  of  the  ETE  Test.) 

JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies  when  using  ADS 
for  T&E? 

JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS  performance  for 
T&E.  This  objective  was  broken  down  into  subobjectives. 

JADS  Subobjective  2-1-1.  Assess  player  instrumentation  and  interface  performance 
constraints  and  concerns.  (This  subobjective  was  not  applicable  to  ETE  Test  Phase  3.) 

JADS  Subobjective  2-1-2.  Assess  network  and  communications  performance  constraints 
and  concerns. 

JADS  Subobjective  2-1-3.  Assess  the  impact  of  ADS  reliability,  availability  and 
maintainability  on  T&E. 

JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support  systems 
for  T&E.  This  objective  was  broken  down  into  subobjectives. 

JADS  Subobjective  2-2-1.  Assess  the  critical  constraints  and  concerns  regarding  ADS 
data  management  and  analysis  systems. 

JADS  Subobjective  2-2-2.  Assess  the  critical  constraints  and  concerns  regarding 
configuration  managment  of  ADS  test  assets.  (This  subobjective  was  not  applicable  to 
Phase  3  of  the  ETE  Test.) 

JADS  Objective  2-3.  Develop  and  assess  methodologies  associated  with  ADS  for  T&E. 
(This  objective  was  not  applicable  to  Phase  3  of  the  ETE  Test.) 

2.4  Phase  3  Methodology 

2.4.1  Tactical  Vignettes 

The  ETE  Test  Phase  3  tactical  vignettes  were  a  subset  of  the  same  vignettes  used  during  Phase  2; 
Northrop  Grumman  used  internal  tapes  from  the  Phase  2  test  and  direct  connections  to  the  ETE 
Test  network. 

The  tactical  vignettes  for  the  ETE  Test  activities  are  unclassified.  The  ETE  Test  team  used  an 
enhanced  TRADOC-approved,  54-hour  corps  battlefield  simulation  (CBS)  scenario  replicating  an 
Iraqi  corps  rear  area  of  operations  in  Iraq.  Five  tactical  vignettes  were  created  in  Janus  6.88D; 


20 


Table  1  provides  a  description  of  each  vignette.  The  following  targets  are  present  in  the  150x150 
kilometer  (km)  Southwest  Asia  (SWA)  terrain  box:  air  defense  artillery  (ADA)  sites,  command 
and  control  sites,  lines  of  communications  (convoys),  logistics  bases,  and  concentrations  of  armor 
and  artillery  units. 


Table  1.  Vignettes  Used  During  ETE  Testing 


Vignette 

Description 

Number  of 
Entities 

1 

Prehostility  phase 

9,897 

2 

Preemptive  strikes 

9,757 

3 

Hammurabi  Division  logistical  operations 

9,904 

4 

Commitment  of  the  Hammurabi  Division 

9,781 

5 

General  headquarters  (GHQ)  depots  to 
corps  and  divisional  logistical  operations 

9,950 

2.4.2  Test  Configuration 

2.4.2. 1  Phase  3  Synthetic  Environment 

ETE  Test  Phase  3  migrated  certain  software  components  of  VSTARS,  specifically  the  ANIU  and 
the  RPSI  from  the  laboratory  Alpha  workstations  to  the  primary  mission  equipment  on  the  T3 
E-8C  aircraft.  In  addition,  the  GNIU  software  was  separated  from  VSTARS  and  migrated  to  an 
Alpha  workstation  collocated  with  a  satellite  transceiver. 

Once  the  migration  was  completed,  each  component  was  tested  in  isolation  and  then  tested  as  a 
part  of  the  complete  environment.  Specifically,  the  network  to  GNIU  link  was  tested  verifying 
that  the  GNIU  was  issuing  a  VSTARS  data  packet  for  each  PDU  received.  The  GNIU  to  satellite 
transceiver  to  satellite  transceiver  to  ANIU  was  also  tested  verifying  that  VSTARS  data  packets 
were  received.  Finally,  the  ANIU  and  RPSI  were  tested  using  primary  mission  equipment  in  the 
laboratory  verifying  that  they  processed  the  data  and  generated  the  appropriate  radar  reports. 
Once  all  components  were  shown  to  be  working,  the  software  was  moved  to  the  aircraft.  The 
entire  environment  was  then  tested  using  PDUs  generated  at  TRAC-WSMR  sent  to  Northrop 
Grumman  and  then  via  satellite  to  the  aircraft. 

2. 4. 2. 2  Phase  3  Testing  at  the  Northrop  Grumman  Node 

Phase  3  testing  at  the  Northrop  Grumman  node  was  conducted  in  four  steps  culminating  with  the 
transition  of  the  ANIU  and  RPSI  to  the  test  aircraft.  Following  this  transition,  a  series  of  system 
integration  tests  (SIT)  were  conducted  by  the  Joint  STARS  Joint  Test  Force,  and  a  complete 
V&V  of  the  RPSI  was  conducted  by  Northrop  Grumman  with  ETE  Test  V&V  team  oversight. 

The  SITs,  conducted  using  the  T3  aircraft  and  a  medium  ground  station  module  (MGSM), 
determined  if  the  software  changes  and  additions  made  to  the  radar  build  in  any  way  affected  the 


21 


performance  of  the  radar  and  operator  workstations.  The  V&V  test  ensured  that  the  ADS 
enhanced  radar  system  met  the  requirements  and  acceptability  criteria  established  by  the  ETE  Test 
team. 

The  Phase  3  test  activities  are  further  discussed  in  section  4. 1  of  this  report. 

2.5  Phase  3  Schedule 


Figure  5  provides  a  schedule  of  the  top-level  tasks  for  Phase  3  of  the  ETE  Test. 


Task  Name 

Dec 

Jan 

Feb 

Mar 

Apr 

May 

Lab  Testing 

Migrate  VSTARS 

to  E-8C  Aircraft 

Ground  Test 

Phase  3  Report 

i 

t 

_ 

* 

Figure  5.  ETE  Test  Schedule 


2.6  Phase  3  Costs 

This  report  does  not  describe  the  costs  of  the  ETE  Test  Phase  3.  Rather,  the  report  on  Phase  4 
will  include  a  work  breakdown  structure  covering  the  costs  of  all  four  phases  of  the  ETE  Test. 
The  Phase  4  report  will  be  published  in  summer  1999. 
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3.0  Phase  3  Execution  Results 

3.1  25-26  February  1999  and  10-12  March  1999  Testing 

Phase  3  testing  took  place  from  25-26  February  1999  and  10-12  March  1999.  Section  4  discusses 
the  specific  measures  addressed  during  this  test  and  the  data  collected  in  support  of  those 
measures. 

3.2  13  March  1999  Testing 

Additional  testing  took  place  at  the  Northrop  Grumman  node  on  13  March  1999.  This  testing 
had  two  objectives. 

•  Allow  the  Joint  STARS  JTF  to  perform  the  formal  system  integration  tests  (SITs)  of  the  Joint 
STARS  aircraft  with  the  JDS  07_006+  software  build  in  operation.  The  SITs  are  designed  to 
verily  the  critical  functionality  of  the  Joint  STARS  system  prior  to  flight  testing. 

•  Conduct  formal  V&V  testing  of  the  JDS  07_006+  software  onboard  the  aircraft. 

The  testing  was  conducted  over  a  12-hour  period  on  13  March  1999.  The  testing  used  recorded 
Janus  vignettes  played  from  equipment  located  in  the  Northrop  Grumman  lab,  then  broadcast  via 
a  satellite  communications  (SATCOM)  link  to  the  aircraft  located  on  the  tarmac.  A  MGSM 
located  at  the  Joint  STARS  JTF  facility  was  used  to  verify  SCDL  linking  ftuictions  with  the 
aircraft. 

All  of  the  formal  testing  conducted  by  the  Joint  STARS  JTF  was  completed  successfully  with  only 
minor  discrepancies  noted.  In  addition,  the  V&V  was  conducted  at  the  same  time.  The  results  of 
these  tests  are  further  discussed  in  section  4.1.1  of  this  report. 
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4.0  Analysis  of  Test  Objectives 


During  Phase  3  of  the  ETE  Test,  JADS  analysts  collected  information  to  address  the  issues, 
objectives,  and  test  measures  as  outlined  in  the  JADS  Program  Test  Plan  (PTP)  and  the  ETE  Test 
Data  Management  and  Analysis  Plan  (DMAP).  Only  those  subobjectives  and  measures  evaluated 
using  Phase  3  results  are  discussed. 

4.1  JADS  Issue  1.  What  is  the  present  utility  of  ADS,  including  DIS,  for  T&E? 

4.1.1  JADS  Objective  1-1.  Assess  the  validity  of  data  from  tests  utilizing  ADS,  including 
DIS,  during  test  execution. 

JADS  Measure  1-1-0-1.  Degree  to  which  ADS  provides  valid  system  under  test  (SUT)  data. 

JADS  Measure  1-1-0-2.  Percentage  of  ADS  data  which  are  valid  (data  supporting  test 
measures  which  are  timely,  accurate,  reliable,  and  otherwise  faithfully  represent  real-world 
systems  data). 

These  two  test  questions  gauge  the  ability  of  an  ADS  environment  to  provide  valid  data  for  the 
C4ISR  SUT.  The  first  measure  addresses  the  validity  of  the  SUT  output  data  which  forms  the 
data  elements  for  evaluating  SUT  measures.  The  second  measure  provides  an  assessment  of  the 
input  data  provided  to  the  SUT  by  the  ADS  environment. 

These  measures  were  addressed  during  Phase  3  by  implementation  of  the  Phase  3  V&V  Plan.  The 
V&V  approach  focused  on  verifying  that  the  changes  made  during  Phase  3  were  compatible  with 
the  ETE  Test  synthetic  environment  (SE).  These  changes  included  the  following: 

•  The  movement  of  the  ANIU  and  the  RPSI  from  the  laboratory  Alpha  workstations  to  the 
primary  mission  equipment  on  the  T3  E-8C  aircraft. 

•  The  migration  of  the  GNIU  software  from  VSTARS  to  an  Alpha  workstation  collocated  with 
a  satellite  transceiver. 

•  The  linking  of  the  GNIU  and  the  ANIU  via  satellite  communications  (SATCOM). 

•  The  replacement  of  the  T- 1  SCDL  with  the  actual  SUT  SCDL. 

These  actions,  in  effect,  replaced  VSTARS  with  an  ADS-enhanced  E-8C  aircraft  within  the  ETE 
Test  SE.  The  remainder  of  the  SE  was  unaffected  by  the  change  because  all  inputs,  outputs,  and 
interactions  were  unchanged.  As  a  result,  all  of  the  V&V  findings  reported  upon  in  the  End-To- 
End  Interim  Report,  Phase  2  still  apply  and  were  not  repeated. 

Phase  3  integration  testing  at  the  Northrop  Grumman  node  was  conducted  in  four  steps 
culminating  with  the  transition  of  the  ANIU  and  RPSI  to  the  test  aircraft.  Following  this 
transition,  a  series  of  SITs  were  conducted  by  the  Joint  STARS  JTF,  and  a  complete  V&V  of  the 
RPSI  was  conducted  by  Northrop  Grumman  with  ETE  Test  V&V  team  oversight.  These  steps 
are  detailed  below  along  with  verification  results. 
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Step  1:  Laboratory  Test  of  the  Isolated  GNIU 

As  stated  above,  the  GNIU  software  component  of  VSTARS  was  modified  to  work  as  an  isolated 
software  component  on  an  Alpha  workstation.  Once  this  was  complete,  an  abbreviated  synthetic 
environment  was  established  to  verily  that  the  GNIU  could  receive  DIS  ESPDUs  and  issue 
corresponding  a  VSTARS  data  packet  (VDP)  for  each  PDU  received.  Figure  6  describes  the 
configuration  for  this  step. 


T-1  I  ine 


DIS  ESPDU  Playback 


GNIU  with  VDP  Logger 


Figure  6.  GNIU  Test  Synthetic  Environment 

In  this  environment,  ESPDUs  that  were  originally  generated  from  a  Janus  6.88D  scenario  were 
played  back  using  a  Silicon  Graphics,  Inc.  (SGI)  JADS  player.  These  ESPDUs  were  broadcast 
over  a  T-1  communications  link  to  the  GNIU.  Upon  receipt  of  the  ESPDUs,  the  GNIU  processed 
the  PDUs  and  issued  correctly  formatted  VDPs. 

Step  2:  Laboratory  Test  of  the  GNIU  to  ANIU  Satellite  Link 

Following  the  testing  of  the  GNIU,  the  next  step  was  to  test  the  GNIU  to  satellite  transceiver  to 
satellite  transceiver  to  ANIU  link.  This  was  accomplished  in  several  stages  starting  with  the 
connection  of  the  satellite  transceivers  in  the  lab  using  a  null  modem  and  culminating  with  the  use 
of  a  communications  satellite  to  transmit  VDPs  from  one  site  to  another  site.  Figure  7  describes 
the  final  configuration  for  this  step.  In  addition  to  developing  and  testing  the  necessary  software 
required  for  the  use  of  the  satellite  transceivers,  this  step  was  also  used  to  develop  the  necessary 
test  tools  needed  to  measure  the  performance  of  the  GNIU  to  ANIU  satellite  link. 


26 


Figure  7.  GNIU  to  ANIU  Link  Synthetic  Environment 

In  this  environment,  ESPDUs  from  a  Janus  6.88D  scenario  were  sent  to  the  GNIU,  converted  to 
VDPs  and  sent  to  a  satellite  transceiver  (Must  Radio)  located  in  one  of  Northrop  Grumman’ s 
laboratories.  The  VDPs  were  then  transmitted  to  a  communications  satellite  and  retransmitted  to 
another  satellite  transceiver  (Must  Radio)  located  in  another  Northrop  Grumman  laboratory.  The 
output  of  the  second  satellite  transceiver  was  recorded  by  a  second  Alpha  workstation 
representing  the  ANIU.  A  review  of  the  recorded  satellite  transceiver  output  showed  an 
acceptable  transmittal  rate  of  34  VDPs  per  second  with  buffering.  At  this  rate,  there  were  no 
dropouts  and  no  corruption  of  the  VDP  packet. 

Step  3:  Laboratory  Test  of  ANIU  and  RPSI  Using  Primary  Mission  Equipment 

The  software  components  of  VSTARS  that  would  be  moved  to  the  T3  E-8C  were  first  moved  to 
the  radar  components  laboratory  (RCL)  and  integrated  into  the  primary  mission  equipment 
(PME).  The  RCL  is  used  to  test  radar  components  and  integrate  software  builds  and  is  a 
duplicate  of  the  equipment  found  on  the  aircraft.  Once  the  software  was  installed  and  integrated, 
it  was  tested  using  the  configuration  shown  in  Figure  8. 
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Building  219 


Building  222 


Figure  8.  Laboratory  PME  Test  of  ANIU  and  RPSI 

The  integration  of  the  ANIU  and  RPSI  into  the  ETE  Test  ADS  software  build  (JDS  07_006+) 
was  tested  by  executing  the  Joint  STARS  JTF  SITs  and  the  Phase  3  V&V  in  the  laboratory  prior 
to  moving  the  integrated  build  to  the  aircraft. 

Step  4:  Test  of  Build  JDS  07_006+  Installed  on  the  T3  E-8C 

Following  the  successful  testing  of  build  JDS  07_006+  in  the  laboratory,  it  was  replicated  and 
installed  on  the  PME  onboard  the  T3  E-8C.  The  test  environment  consisted  of  ESPDUs  from  a 
Janus  6.88D  scenario  transmitted  from  a  SGI  JADS  player  or  TRAC-WSMR  to  the  ground  NIU 
via  a  T-l  communications  line.  The  ESPDUs  were  then  converted  to  VDPs  and  transmitted  to 
the  air  NIU  on  the  E-8C  via  a  SATCOM  link.  These  data  were  then  used  by  the  RPSI  to  generate 
virtual  radar  reports,  which  were  mixed  with  live  radar  reports  from  noise  generated  by  a  dummy 
load  and  sent  out  on  the  OWS  local  area  network  (LAN)  to  the  workstations  and  via  an  actual 
SCDL  to  a  ground  station  module  sitting  several  hundred  meters  from  the  aircraft.  Figure  9 
depicts  the  configuration  for  this  step. 


28 


Figure  9.  Synthetic  Environment  for  Test  of  Build  JDS  07_006+  Installed  on  the  T3  E-8C 

The  SATCOM  link  between  the  GNIU  and  the  ANIU  was  tested  and  characterized  by  Northrop 
Grumman  and  proper  operation  was  verified. 

Once  it  was  ascertained  that  build  JDS  07_006+  appeared  to  be  functioning  correctly,  it  was 
tested  by  the  Joint  STARS  JTF  executing  the  required  SITs  and  Northrop  Grumman  personnel 
accomplishing  the  Phase  3  V&V. 

System  Integration  Tests 

The  Joint  STARS  JTF  required,  prior  to  any  test  flight,  that  a  series  of  SITs  be  conducted  using 
the  software  build  that  would  be  flown  during  the  flight.  The  SITs  ensured  the  ability  to  use  the 
subsystems  onboard  the  aircraft  (radar,  advanced  tactical  workstations,  communications,  and 
SCDL)  was  not  compromised  in  any  way  by  the  software  changes  and  additions  made  to  the  radar 
build.  The  SITs  were  conducted  using  the  T3  aircraft  and  an  MGSM.  V&V  was  conducted  to 
ensure  that  the  ADS-enhanced  radar  system  met  the  requirements  and  acceptability  criteria 
established  by  the  ETE  Test  team. 
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The  results  from  implementing  the  ETE  Test  Phase  3  V&V  are  detailed  in  the  Phase  3  V&V 
reports  and  are  summarized  as  follows.  (ETE  Test  W&A  represented  a  tailoring  and 
implementation  of  the  nine-step  process  to  a  multiservice  test  of  a  major  system.  Joint  STARS, 
augmented  with  ADS.  The  tailored  ETE  Test  process  model  used  is  described  in  the  V&V 
reports  for  the  ETE  Test.  Available  from  JADS,  2050A  2nd  Street  SE,  Kirtland  Air  Force  Base, 
New  Mexico,  87117-5522.  After  1  March  2000  refer  requests  to  HQ  AFOTEC/HO,  8500 
Gibson  Blvd  SE,  Kirtland  Air  Force  Base,  New  Mexico  87117-5558,  or  SAIC  Technical  Library, 
2001  North  Beauregard  St.  Suite  80,  Alexandria,  Virginia  22311.) 

•  Verification  of  the  ADS-enhanced  E-8C  aircraft 

-  The  following  were  verified  during  the  SITs 

•  JDS  07_006+  permitted  all  of  the  aircraft  subsystems  to  function  normally 

•  JDS  07_006+  processed  parameter  data  in  the  same  format  as  Joint  STARS 

•  JDS  07_006+  permitted  all  of  the  installed  operator  workstation  software  to  function 
without  abnormal  fault  messages  occurring 

•  JDS  07_006+  received  and  integrated  virtual  data  from  the  ADS  environment 

•  JDS  07_006+  operated  in  three  modes:  live  only,  mixed  live  and  virtual,  and  virtual 
only  using  the  standard  Joint  STARS  MTI  message  format 

•  The  radar  timeline  was  not  impacted  by  the  MTI  simulation 

-  The  requirement  that  JDS  07_006+  display  live  SARs  in  live  areas  of  interest  and  virtual 
SARs  in  both  virtual  and  mixed  areas  of  interest  using  the  standard  Joint  STARS  SAR 
message  format  was  not  completely  met.  The  software  build  contained  an  error, 
previously  observed  and  corrected  in  VSTARS,  that  resulted  in  live  SARs  displayed  in  a 
mixed  area  of  interest  (in  which  only  virtual  SARs  should  have  been  displayed). 
Correction  of  this  fault,  though  relatively  easy,  would  have  required  a  redo  of  both  the 
SITs  and  that  portion  of  the  verification.  Since  the  aircraft  would  not  be  available  before 
the  first  flight,  nor  between  the  subsequent  flights,  for  this  additional  testing,  and  the 
shortcoming  would  have  no  impact  on  the  operational  test,  the  decision  was  made  to 
proceed  with  the  test  without  this  capability. 

-  There  was  a  problem  with  corruption  of  the  data  packets  when  sent  via  the  satellite  link. 
This  problem  manifested  itself  by  identifying  nonmoving  targets  as  moving  targets.  One  of 
the  programmers  had  found  it  necessary  to  add  thirty-two  bits  to  the  VSTARS  data 
packet  in  order  to  separate  the  GNIU  and  the  ANIU.  The  programmer  working  on  the 
satellite  link  was  not  told  this  and  continued  to  parse  the  data  packets  as  192-bit  as 
opposed  to  224-bit  data  packets.  Once  the  error  was  found,  it  was  corrected  and  that 
portion  of  the  V&V  was  repeated  prior  to  the  Phase  4  flight  tests. 

•  Verification  of  the  SCDL 

-  The  SCDL  was  tested  by  the  Joint  STARS  JTF  during  the  conduct  of  the  SITs  onboard 
the  aircraft. 

-  The  aircraft  was  linked  to  both  the  SCDL  laboratory  at  Northrop  Grumman  and  to  a 
LGSM  that  belonged  to  the  Joint  STARS  JTF.  During  this  testing,  it  was  verified  that 
JDS  07_006+  could  link  to  both  the  old  SCDL  format  and  the  new  SCDL  format  allowing 
its  use  with  both  GSMs  and  common  ground  stations  (CGSs). 
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•  Validation  of  JDS  07_006+.  The  validation  of  JDS  07_006+  was  performed  by  the  Joint 
STARS  JTF  operators  who  performed  the  SITs  and  included  several  of  the  operators  who 
took  part  in  the  Phase  2  validation  of  VSTARS.  It  also  included  several  operators  who  had 
not  previously  seen  ADS-enhanced  radar. 

—  All  of  the  operators  were  impressed  with  the  performance  of  JDS  07_006+,  and  those  that 
had  previously  tested  VSTARS  noticed  no  differences  from  the  previously  validated 
laboratory  version.  The  operators  that  had  not  previously  seen  ADS-enhanced  radar  made 
the  same  comments  as  noted  in  the  Phase  2  V&V  report. 

Conclusion  for  JADS  Measure  1-1-0-1.  The  Phase  3  ADS  configuration  produced  more  valid 
data  than  the  Phase  2  configuration.  This  was  because  of  the  increased  use  of  actual  processes 
and  hardware.  All  simulation  processes  were  functioning  on  actual  SUT  hardware  using  standard 
processes  to  include  SCDL.  The  only  simulations  occurring  were  the  simulations  of  the  MTI, 
SAR  and  fixed  target  indicator  (FTI)  modes.  These  simulations  ran  parallel  to  the  actual  radar 
using  its  timelines  and  output  standard  radar  reports.  These  reports  were  then  mixed  with  the 
actual  radar  reports,  as  designated  by  the  simulation  manager,  into  five,  virtual,  and  mixed  radar 
reports.  Events  that  degrade  the  quality  of  the  data  do  occur,  such  as  LAN  collisions,  but  they 
occur  equally  to  both  real  and  simulated  data. 

Conclusion  for  JADS  Measure  1-1-0-2.  Under  normal  operations,  all  input  data  provided  to  the 
SUT  (Joint  STARS)  by  the  ADS  environment  were  valid.  Network  performance  and  reliability  in 
delivering  data  to  the  SUT  are  analyzed  under  JADS  Objective  2-1. 

Utility  for  OT&E 

The  utility  of  this  configuration  for  Joint  STARS  OT&E  was  evaluated  by  determining  which 
measures  from  the  Joint  STARS  Multiservice  Operational  Test  and  Evaluation  (MOT&E)  Plan1 
could  be  supported  assuming  the  use  of  the  E-8C  aircraft  on  the  tarmac,  fully  manned  and  linked 
to  a  fully  manned  LGSM.  Appendix  B  (available  under  separate  cover  from  JADS  JTF)  identifies 
which  Joint  STARS  MOT&E  measures  could  be  evaluated  using  the  Phase  3  ADS  configuration. 

Results  in  Appendix  B  are  summarized  as  follows. 

-  The  measures  for  critical  operational  issue  (COI)-l  (Does  Joint  STARS  perform  its  tactical 
battlefield  surveillance  mission?)  involving  the  performance  of  the  E-8  radar  in  its  operational 
environment  cannot  be  evaluated  using  the  Phase  3  configuration.  As  a  result,  the  Phase  3 
configuration  could  be  used  to  evaluate  only  7  out  of  18  measures  of  performance  (MOPs) 
supporting  COI-1.  However,  all  three  measures  of  effectiveness  (MOEs)  for  COI-1  could  at 
least  be  partially  evaluated  using  the  Phase  2  configuration. 


1  Joint  Surveillance  Target  Attack  Radar  System  (Joint  STARS)  Multiservice  Operational  Test  and  Evaluation 
(MOT&E)  Plan,  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base,  New  Mexico,  21 
February  1995. 
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-  As  with  COI-1,  the  measures  for  COI-2  (Does  Joint  STARS  support  the  execution  of  attacks 
against  detected  targets?)  involving  the  performance  of  the  E-8  radar  in  its  operational 
environment  cannot  be  evaluated  using  the  Phase  3  configuration.  As  a  result,  the  Phase  3 
configuration  could  be  used  to  evaluate  only  1 1  out  of  24  MOPs,  resulting  in  a  very  limited 
evaluation  for  3  out  of  3  MOEs  supporting  COI-2. 

-  The  Phase  3  ADS  configuration  could  be  used  to  partially  evaluate  the  single  MOE  supporting 
COI-3  (Does  Joint  STARS  provide  timely  and  accurate  information  to  support  battlefield 
management  and  target  selection?). 

-  The  Phase  3  ADS  configuration  could  be  used  to  evaluate  1  out  of  2  MOPs  for  COI-4  (Can 
the  Joint  STARS  system  be  sustained  in  an  operational  environment?). 

-  The  Phase  3  ADS  configuration  could  support  the  evaluation  of  8  out  of  17  of  the  additional 
effectiveness  measures. 

-  The  Phase  3  ADS  configuration  could  support  the  evaluation  of  13  out  of  27  MOPs  involving 
GSM  or  E-8C  suitability. 

-  The  Phase  3  configuration  could  allow  for  operational  operators  to  be  used  which  would 
allow  for  all  8  of  the  E-8C  human  factors  measures  to  be  addressed. 

-  The  Phase  3  configuration  could  allow  for  evaluation  of  6  out  of  6  software  system  MOPs. 

In  summary,  the  Phase  3  ADS  configuration  could  only  allow  an  evaluation  of  22  out  of  45 
effectiveness  MOPs  and  a  very  limited  evaluation  for  7  of  8  effectiveness  MOEs.  Further,  the 
Phase  3  ADS  configuration  could  be  used  to  evaluate  the  GSM  measures  (13  out  of  27  suitability 
MOPs),  all  8  of  the  E-8C  human  factors  MOPs  and  all  6  software  MOPs. 

4.2  JADS  Issue  2.  What  are  the  critical  constraints,  concerns,  and  methodologies 
when  using  ADS  for  T&E? 

The  evaluation  of  this  issue  was  based  on  testing  using  the  connectivity  test  configuration  (T-l 
lines  only;  no  SATCOM  link),  rather  than  the  Phase  3  ADS  configuration  (with  SATCOM  link). 

4.2.1  JADS  Objective  2-1.  Assess  the  critical  constraints  and  concerns  in  ADS 
performance  for  T&E. 

4.2. 1. 1  JADS  Subobjective  2-1-2.  Assess  network  and  communications  performance  constraints 
and  concerns. 

JADS  Measure  2-1-2-2.  Percentage  of  ADS  trials  canceled  or  otherwise  not  used  due  to 
network  problems. 
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JADS  Measure  2-1-3-3.  Percentage  of  trials  in  which  network  connections  were  lost  long 
enough  to  require  trial  cancellation. 

For  these  measures,  the  network  was  defined  as  including  all  software  and  hardware  used  for 
connecting  the  distributed  sites  and  all  loggers  and  instrumentation  used  for  recording  network 
data.  NIUs  were  considered  part  of  the  individual  simulations  and  not  part  of  the  network. 

For  each  trial,  an  execution  log  was  maintained  at  each  node.  The  data  collectors  annotated  all 
problems  encountered,  including  loss  of  connectivity  in  any  link,  as  well  as  their  causes.  A  test 
controller  log  also  documented  the  overall  status  of  the  network  and  test  trials.  In  addition, 
network  monitoring  tools  were  used  to  monitor  the  status  of  all  network  links  between  nodes. 
Any  problems  detected  by  the  monitoring  tools  were  documented  via  line  printers  in  terms  of  a 
brief  explanation  of  the  problem,  the  time,  and  the  link(s)  involved. 

There  were  no  network  problems  which  were  serious  enough  to  require  delay  or  cancellation  of 
any  trials.  Although  there  were  some  losses  of  connectivity  (see  Measure  2- 1-3-6),  these  were  of 
short  duration  and  did  not  significantly  impact  the  trials.  During  all  previous  ETE  Test  phases, 
significant  network  problems  were  experienced  for  at  least  a  portion  of  the  tests.  The  risk 
reduction  efforts  taken  during  these  previous  test  phases  helped  to  ensure  the  reliability  of  the 
network  during  the  Phase  3  connectivity  tests.  Table  2  shows  the  dates  of  the  trials  and  their  test 
times.  Note  that  there  were  some  trial  cancellations  because  of  ADS  system  failures  (see 
Measures  2-1-3-1  and  2-1-3-5)  exclusive  of  network  problems. 

Table  2.  ETE  Test  Phase  3  Connectivity  Tests 


Trial 

Time 
Scheduled 
for  Testing 

Comments 

25  February 

Trial  canceled  because  of  VSTARS  unavailability 

Trial  canceled  because  of  VSTARS  unavailability 

10  March 

Trial  delayed  and  then  canceled  because  of  VSTARS 
unavailability 

1 1  March 

7  hrs,  37  mins 

12  March 

6  hrs,  36  mins 

JADS  Measure  2-1-2-3.  Bandwidth  utilized  and  packet  rate  by  link. 

This  measure  provided  an  indication  of  bandwidth  use  and  packet  rate  during  the  Phase  3 
connectivity  tests.  Although  bandwidth  utilization  was  not  expected  to  exceed  capacity,  the 
utilization  rate  was  documented  to  provide  other  ADS  testers  with  an  indication  of  the  amount  of 
needed  bandwidth.  The  packet  rate  data  are  also  included  because  of  their  potential  value  to 
other  ADS  testers. 

Data  were  collected  using  the  Spectrum™  network  analysis  tool.  Spectrum™  provided  the 
capability  to  study  multiple  aspects  of  network  link  performance  including  packet  rate  and 
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percentage  of  bandwidth  utilized.  A  polling  rate  of  five  seconds  was  used  in  the  collection  of 
these  data. 

Once  all  the  data  were  gathered,  the  JADS  analysts  consolidated  the  data  by  network  link.  These 
data  were  then  used  to  calculate  daily  packet  rate  and  bandwidth  values  (maximum  and  average) 
for  each  link.  The  bandwidth  values  were  provided  by  Spectrum™  as  the  percentage  of 
bandwidth  available  on  the  T-l  line.  A  T-l  line  has  a  normal  bandwidth  of  1.544  megabits  per 
second  (Mbps).  For  the  ETE  Test,  some  of  the  bandwidth  of  the  T-l  line  was  reserved  for  voice 
traffic,  leaving  a  maximum  bandwidth  available  of  1.344  Mbps. 

Table  3  shows  average  and  maximum  performance  values  for  the  classified  network  links 
monitored  during  the  two  days  of  active  Phase  3  connectivity  testing. 

Table  3.  Connectivity  Tests  Link  Performance  Characteristics* 


Day 

Node  A 

Node  B 

Load 

Average  Maximum 

Packet  Rate 

Average  Maximum 

11  March 

T 

G 

1.4% 

73% 

18.8/sec 

331/sec 

G 

H 

0.1% 

4% 

6.2/sec 

87/sec 

H 

T 

0.05% 

3% 

3.2/sec 

14/sec 

12  March 

T 

G 

0.05% 

1% 

2.5/sec 

19/sec 

G 

H 

1.0% 

12% 

24.3/sec 

119/sec 

H 

T 

2.5% 

19% 

15.9/sec 

54/sec 

T  =  TCAC 


G  = 


Northrop  Grumman 


H  =  Fort  Hood 


*  Table  refers  only  to  active  test  time  during  which  PDU  loggers  were  recording  data. 


Packet  rate  and  bandwidth  utilized  differed  greatly  between  the  two  days  of  testing  because  of  a 
cryptographic  equipment  problem  which  impacted  the  TCAC-Grumman  link  prior  to  test  start  on 
12  March.  The  link  was  not  used  to  pass  data  traffic  for  about  five  out  of  the  six  hours  of  testing 
that  day,  during  which  time  the  data  were  automatically  rerouted  between  the  two  sites  using  the 
alternate  network  path  through  Fort  Hood  until  the  problem  was  fixed.  Although  the  flow  of  test 
data  along  this  alternate  path  was  transparent  to  the  testers,  analyzed  packet  rate  and  bandwidth 
data  for  the  three  network  links  were  quite  different  between  the  two  days.  The  packet  rate 
experienced  over  the  TCAC-Grumman  link  averaged  approximately  19  packets  per  second  on  1 1 
March  but  only  2.5  on  12  March.  The  packet  rate  experienced  over  the  Grumman-Fort  Hood  link 
jumped  from  an  average  of  6.2  packets  per  second  on  11  March  to  24.3  on  12  March,  and  the 
Fort  Hood-TCAC  traffic  jumped  from  an  average  of  3.2  packets  per  second  on  1 1  March  to 
about  16  on  12  March.  The  latter  two  links  took  up  the  data  traffic  responsibility  of  the  downed 
link,  and  testing  continued  without  a  hitch  proving  the  utility  of  network  link  redundancy.  The 
maximum  packet  rate  for  any  network  link  during  the  two  test  days  was  33 1  packets  per  second, 
experienced  over  the  TCAC-Grumman  link  on  1 1  March,  resulting  in  a  peak  load  of  73  percent  of 
bandwidth  capacity.  This  high  rate  was  of  short  duration  (about  2  minutes)  and  appeared  to  be 
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caused  by  file  transfers  during  the  trial  rather  than  by  normal  PDU  traffic.  The  greatest  average 
load  experienced  across  any  of  the  three  network  links  was  2.5  percent,  showing  the  relatively 
small  bandwidth  utilization  experienced  during  the  Phase  3  connectivity  test. 

It  is  also  noted  that  the  average  packet  rates  and  bandwidth  utilization  rates  measured  on  1 1 
March  were  consistent  with  values  from  the  Phase  2  trials,  showing  the  relative  stability  of  the 
network. 

JADS  Measure  2-1-2-5.  Percentage  of  time  PDUs  were  received  out  of  order  by  a  network 
node. 

JADS  Measure  2-1-2-6.  Percentage  of  total  PDUs  required  at  a  node  that  were  delivered  to 
that  node. 

JADS  Measure  2-1-2-7.  Average  and  peak  data  latency  between  ADS  nodes. 

The  flow  of  PDUs  to  and  from  each  node  was  recorded  using  loggers  installed  as  part  of  the 
network  architecture.  The  loggers  specifically  recorded  the  time  and  order  that  the  PDUs  were 
transmitted  and  received  at  each  node. 

The  raw  logger  data  were  transformed  and  reduced  for  analysis  to  determine  out  of  order, 
duplicate  or  lost  PDUs  and  PDU  latency.  These  data  were  then  used  to  calculate  the  percentage 
of  out  of  order,  duplicate,  and  lost  PDUs  at  each  node  for  each  test  day  and  for  the  connectivity 
test  as  a  whole.  The  minimum,  maximum,  and  mean  latency  of  PDUs  were  also  computed.  JADS 
analysts  accomplished  these  calculations  using  UNIX®-based  software  tools  created  by  JADS 
programmers. 

Table  4  shows  the  PDU  data  for  each  day  by  node;  there  were  no  duplicate  or  out  of  order  PDUs. 
The  PDU  data  in  Table  4  show  total  PDU  loss  rates  of  2.75,  0.48,  and  1.71  percent  for  the 
WSMR-TCAC,  TCAC-Northrop  Grumman,  and  Fort  Sill-WSMR  links,  respectively.  Note  that 
the  total  loss  rate  for  ESPDUs  generated  by  Janus  being  delivered  from  the  WSMR  node  to  the 
Northrop  Grumman  node  (the  node  requiring  them)  was  3.22  percent  (or  7,281  PDUs  lost  out  of 
226,440  PDUs  sent). 

The  overall  loss  rate  between  WSMR  and  Northrop  Grumman  and  between  Fort  Sill  and  WSMR 
was  3.18  percent.  This  PDU  loss  rate,  while  still  well  under  the  criterion  of  not  using  trial  data 
with  5  percent  or  more  lost  PDUs,  is  considerably  higher  than  the  loss  rate  experienced  during  the 
Phase  2  test  (0.075  percent).  This  resulted  in  large  part  from  the  PDU  losses  caused  by  the 
temporary  outages  of  the  WSMR-TCAC  and  Fort  Sill-WSMR  links  on  12  March,  as  shown  in 
Table  5.  Table  5  gives  the  estimated  PDU  losses  because  of  the  loss  of  network  link  connectivity 
(estimated  by  correlating  PDU  time  stamps  with  link  outage  times)  and  shows  that  80  to  90 
percent  of  the  PDU  losses  were  due  to  link  connectivity  losses.  The  overall  loss  rate  because  of 
causes  other  than  link  connectivity  losses  was  about  0.7  percent  which  is  much  more  consistent 
with  the  Phase  2  losses. 
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Table  4.  Connectivity  Tests  PDU  Data 


Date 

Node  A 

NodeB 

PDUs 

Sent 

PDUs 

Received 

PDUs  Lost/ 
Percent  Lost 

w 

T 

93,896 

93,872 

24 

0.03% 

11  March 

T 

G 

93,872 

92,837 

1,035 

1.10% 

S 

W 

2,618 

2,617 

1 

0.04% 

w 

T 

132,544 

126,338 

6,206 

4.68% 

12  March 

T 

G 

126,338 

126,322 

16 

0.013% 

S 

W 

2,455 

2,369 

86 

3.50% 

w 

T 

226,440 

220,210 

6,230 

2.75% 

Total 

T 

G 

220,210 

219,159 

1,051 

0.477% 

S 

w 

5,073 

4,986 

87 

1.71% 

W=  WSMR  T  =  TCAC 


G  =  Northrop  Grumman 


S  =  Fort  Sill 


Table  6  shows  the  latencies  measured  during  the  Phase  3  connectivity  tests.  These  data  show  that 
the  average  latency  over  the  Fort  Sill-WSMR  link  was  very  stable  during  the  two  days  of  testing 
and  was  within  10  percent  of  the  Phase  2  value.  The  average  latency  over  the  WSMR-TCAC  link 
was  not  nearly  as  stable.  The  WSMR-TCAC  link  average  on  11  March  was  within  about  5 
percent  of  the  Phase  2  value,  but  the  average  on  12  March  was  significantly  higher.  The  latter 
value  may  be  uncharacteristically  high  because  of  network  problems  with  this  link  on  that  day. 

As  for  the  TCAC-Grumman  link,  no  comparison  could  be  made  between  the  Phase  2  and  Phase  3 
latencies  because  of  the  Phase  3  connectivity  tests’  time  synchronization  problem  which  resulted 
in  negative  (i.e.,  invalid)  latencies.  This  problem  was  resolved  for  the  Phase  4  test. 
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Table  5.  Connectivity  Tests  PDU  Losses  Due  to  Network  Link  Losses 


Date 

Node 

A 

Node 

B 

PDUs 

Sent 

PDUs  Lost  Due  to 
Network  Link  Loss 

PDUs 

Lost 

%  of  PDUs  Sent  / 

%  of  PDUs  Lost 

11  March 

W 

T 

93,896 

24 

0 

0%  /  0% 

H 

G 

876 

.93%  /  84.64% 

s 

0 

0%  /  0% 

12  March 

w 

T 

4,822 

3.64%  /  77.70% 

T 

G 

0 

0%  /  0% 

S 

W 

2,455 

86 

80 

3.26%  /  93.02% 

Total 

w 

T 

IBM 

4,822 

2.13%  /  77.40% 

876 

.40%  /  83.35% 

5,073 

87 

80 

1.58%  7  91.95% 

W=  WSMR  T  =  TCAC  G  =  Northrop  Grumman  S  =  Fort  Sill 


Table  6.  Connectivity  Tests  Latency  Data 


Date 

Node  A 

NodeB 

Latency  (seconds)  "1 

Minimum 

Mean 

Maximum 

11  March 

W 

T 

0.020 

0.129 

T 

G 

_ * 

_ * 

S 

W 

0.037 

0.038 

0.375 

12  March 

w 

T 

0.020 

0.053 

0.172 

T 

G 

S 

W 

0.036 

0.038 

0.365 

Total 

W 

T 

0.020 

0.047 

0.172 

T 

G 

—  —  * 

_* 

_ * 

S 

W 

0.036 

0.038 

0.375 

W=  WSMR  T  =  TCAC  G  =  Northrop  Grumman  S  =  Fort  Sill 

*  Logger  clocks  could  not  be  synchronized  at  the  Grumman  node  because  of  a 
problem  with  the  time  synchronization  program.  This  problem  resulted  in  negative 
latencies.  However,  the  problem  was  resolved  following  testing  on  12  March. 
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4. 2. 1.2  JADS  Subobjective  2-1-3.  Assess  the  impact  of  ADS  reliability,  availability  and 
maintainability  on  T&E. 

Intent.  This  subobjective  examined  the  ability  of  the  ADS  systems  (players  and  network)  to  be 
up  and  operating  at  scheduled  test  initialization  and  to  remain  up  and  operating  throughout  the 
duration  of  the  test. 

JADS  Measure  2-1-3-1.  Number  of  trials  delayed,  rescheduled,  and/or  reaccomplished 
because  of  failure  of  ADS  systems,  exclusive  of  network  unavailability. 

JADS  Measure  2-1-3-5.  Number  of  ADS  system  failures. 

These  measures  determined  the  availability  of  ADS  nodes  including  the  NIUs  and  the  impact  of 
node  failures  on  Phase  3  testing. 

For  each  trial,  an  execution  log  was  maintained  at  each  node.  The  data  collectors  annotated  all 
problems  encountered  with  the  ADS  systems  along  with  their  causes.  A  test  controller  log  was 
also  maintained  to  document  the  overall  status  of  the  trials. 

A  total  of  seven  ADS  system  failures  occurred  during  the  Phase  3  connectivity  tests.  Six  of  the 
seven  failures  involved  VSTARS,  with  the  other  ADS  system  failure  due  to  TAFSM.  While  the 
TAFSM  failure  resulted  in  only  a  3-minute  delay  in  running  TAFSM  and  no  impact  on  the  overall 
trial,  the  VSTARS  failures  resulted  in  the  cancellation  of  three  test  trials  and  the  degradation  of 
the  SCDL  during  the  remaining  two  trials. 

The  SCDL  between  the  LGSM  and  VSTARS  did  not  function  properly  during  any  of  the  Phase  3 
connectivity  tests.  The  LGSM  at  Fort  Hood  could  send  messages  to  VSTARS  but  received  only 
garbled  text  messages  and  imagery.  The  SCDL  failure  was  due  to  attempting  to  use  the  version 
of  the  RPSI  developed  for  the  aircraft  on  the  laboratory  Alpha  workstations.  This  was  attempted 
because  the  necessary  offsets  had  been  applied  to  this  software,  and  it  was  desirable  to  verify  that 
the  offsets  worked  correctly  with  the  Fort  Hood  GSM.  Once  it  was  determined  that  the  JDS 
07_006+  version  of  the  RPSI  would  not  work  properly  on  an  Alpha  workstation,  the  original 
RPSI  used  in  the  Phase  2  test  was  resurrected  and,  after  the  proper  offsets  were  applied,  used  for 
the  Phase  4  testing. 

Table  7  lists  the  reported  ADS  failures,  along  with  the  time  needed  to  resolve  these  interruptions 
and  their  impact  on  testing. 
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Table  7.  ADS  System  Failures 


Day 

Failure 

Test 

Time 

Impact  on  Test 

25 

February 

VSTARS  not 
operational 

Northrop 
Grumman 
unable  to 
resolve  for 
trial 

N/A 

N/A 

Trial  canceled 

26 

February 

VSTARS  not 
operational 

Northrop 
Grumman 
unable  to 
resolve  for 
trial 

N/A 

N/A 

Trial  canceled 

10  March 

VSTARS  not 
operational 

Software 
adjustments 
for  lab 
environment 

N/A 

.  N/A 

Test  startup  delayed 
pending  VSTARS 
fix 

SCDL  not 
operational 

Northrop 
Grumman 
unable  to 
resolve  for 
trial 

N/A 

N/A 

Trial  canceled 

11  March 

TAFSM 
crashed  at 

Fort  Sill 

TAFSM 

rebooted 

3  mins 

7  hrs, 

37  mins 

No  delay  caused  by 
TAFSM  reboot 

SCDL  not 
operational 

LGSM 

rebooted; 

VSTARS 

SCDL 

checked; 

problem 

unresolved 

6  hrs, 

5  mins 

7  hrs, 

37  mins 

Fort  Hood  received 
garbled  imagery  and 
messages  via  the 
SCDL  for  the  entire 
trial 

12  March 

SCDL  not 
operational 

Northrop 
Grumman 
adjusted 
SCDL  on 
VSTARS 

6  hrs, 

14  mins 

6  hrs, 

36  mins 

Fort  Hood  received 
garbled  imagery  and 
messages  via  the 
SCDL  for  the  entire 
trial 
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JADS  Measure  2-1-3-6.  Average  down  time  due  to  ADS  network  failures. 

This  measure  identified  the  impact  of  network  failures  on  the  Phase  3  test.  During  Phase  3,  logs 
were  kept  to  record  all  network  problems,  the  start  time  and  duration  of  the  problems  and 
problem  resolution.  In  addition,  network  monitoring  tools  were  used  to  monitor  the  status  of  all 
network  links  between  the  nodes.  Any  problem  detected  by  the  monitoring  tools  was 
documented  via  line  printers  in  terms  of  a  brief  explanation  of  the  problem,  the  time,  and  the 
link(s)  involved. 

Because  of  problems  with  VSTARS,  not  the  network,  the  first  two  scheduled  trials  were  not 
executed,  and  the  third  trial  was  not  completed.  For  the  two  trials  that  were  accomplished,  only 
three  network  outages  were  experienced,  resulting  in  a  total  of  1 1  minutes  of  network  downtime 
during  the  Phase  3  connectivity  tests.  Table  8  displays  the  data  on  network  downtime. 

Table  8.  Network  Downtime 


Date 

Time 

Scheduled  for 
Testing 

Time  Network 
Unavailable 
for  Testing 

Percentage 
of  Time 
Network 
Unavailable 

Reason  Unavailable 

11  March 

7  hrs,  37  min 

6  mins 

1.31% 

Router  down  at  Northrop 
Grumman;  unknown  problem  at 
Northrop  Grumman 

12  March 

6  hrs,  36  mins 

5  mins 

1.26% 

Unknown  problem  at  WSMR 

Total 

14  hrs,  13  mins 

11  mins 

1.29% 

Table  8  shows  that  the  network  was  reliable  during  the  execution  of  the  Phase  3  connectivity 
tests.  During  the  two  days  of  testing,  the  network  was  down  for  only  1 1  minutes  or  1.29  percent 
of  the  test  period  resulting  in  few  lost  (2.5%)  PDUs.  The  causes  of  two  of  the  three  documented 
network  problems  were  unknown.  There  was  a  problem  experienced  at  the  Northrop  Grumman 
node  on  1 1  March  that  resolved  itself  before  JADS  N&E  personnel  could  attempt  to  identify  it. 
There  was  also  an  unidentified  problem  at  WSMR  on  12  March.  This  problem  was  examined  by 
JADS  N&E  personnel  but  could  not  be  readily  identified.  It  is  most  likely  that  this  problem  was 
due  to  the  severe  weather  experienced  at  WSMR  which  affected  networks  post  wide.  Again,  this 
problem  resolved  itself  and  is  not  expected  to  be  a  factor  during  Phase  4  testing. 

4.2.2  JADS  Objective  2-2.  Assess  the  critical  constraints  and  concerns  in  ADS  support 
systems  for  T&E. 

4.2.2. 1  JADS  Subobjective  2-2-1.  Assess  the  critical  constraints  and  concerns  regarding  ADS 
data  management  and  analysis  systems. 

JADS  Measure  2-2-1-1.  Degree  to  which  ADS  nodes  provide  for  collection,  data  entry,  and 
quality  checking  of  pre-  and  post-trial  briefing  data. 
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Quick-look  analysis  of  results  was  used  to  support  the  post-trial  briefings.  This  analysis  relied 
primarily  on  automated  data  collection  at  all  ETE  Test  nodes.  The  data  collection  tools  included 
the  JADS  logger  which  collected  the  PDU  log  files  and  a  Spectrum™  logger  to  monitor  network 
performance.  Data  collection  tools  were  attached  to  the  network  at  each  node  without  any 
impact  on  network  or  node  performance.  At  the  end  of  each  test  day,  the  data  were  remotely 
retrieved  by  the  TCAC  and  the  file  size  checked.  This  procedure  supported  timely  quick-look 
analysis  and  test  feedback. 

In  addition  to  electronic  data  logs,  manually  written  logs  were  kept  at  each  test  site  and  used  to 
support  post-trial  briefings.  In  addition,  a  daily  after-action  teleconference  call  was  added.  This 
enabled  the  test  controller  to  discuss  and  fully  understand  the  problems  of  the  day  without  having 
to  review  local  log  sheets. 

JADS  Measure  2-2-1-2.  Adequacy  of  relevant  test  data  storage  at  ADS  nodes. 

The  ETE  Test  analysis  requirements  drove  test  data  storage  needs.  The  focus  of  data  analysis  at 
each  site  was  on  network  latency,  as  well  as  the  actual  PDU  input  or  data  output  at  each  site.  The 
need  to  record  PDU  traffic  at  each  node  required  a  determination  of  the  data  output  and  reception 
rates  at  all  sites.  The  largest  contributor  to  ESPDU  traffic  was  the  output  of  the  Janus  simulation. 
ESPDUs  from  Janus  are  a  function  of  the  Janus  heartbeat  and  the  vignette  design.  During  the 
Phase  3  testing,  the  Janus  heartbeat  was  set  to  update  all  entities  every  eleven  minutes  during  the 
first  hour.  In  addition,  Janus  had  to  output  an  ESPDU  when  an  entity  changed  state,  i.e.,  start, 
stop,  turn,  etc.  As  a  result,  the  ESPDU  output  grew  as  the  number  of  movers  increased.  The 
ETE  Test  used  five  different  vignettes,  ranging  from  prehostility  with  low  numbers  of  movers  to 
an  active  battle  vignette  with  more  than  3,000  entities  moving  at  one  time.  Prior  to  Phase  2 
testing,  the  five  vignettes  were  played  and  the  ESPDU  output  recorded.  The  maximum  file  size 
during  this  testing  was  about  fifteen  megabytes.  To  support  the  data  recording  as  well  as  file 
storage  and  local  software  requirements,  the  JADS  N&E  team  installed  4-gigabyte  hard  drives  on 
the  SGI  Indy  at  each  node. 

During  preparations  for  the  Phase  3  test,  the  Northrop  Grumman  node  required  the  largest  data 
capacity  in  order  to  support  VSTARS  software  testing  in  a  stand-alone  mode.  This  testing 
required  the  playback  of  PDU  files  recorded  from  TRAC-WSMR  to  VSTARS.  All  five  vignettes 
were  played  back  at  various  times,  and  at  least  five  vignette  PDU  files  were  stored  on  the  SGI 
Indy  at  all  times.  During  actual  Phase  3  testing,  the  ETE  Test  team  found  hard  drive  data  storage 
capacity  to  be  more  than  adequate. 

The  development  of  data  storage  needs  required  a  full  understanding  of  each  node’s  requirements. 
Since  the  cost  of  hard  drive  storage  has  decreased  dramatically  over  the  past  few  years,  it  was 
cost  effective  to  allow  for  unexpected  growth  by  significantly  exceeding  the  expected  storage 
requirements. 
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5.0  Lessons  Learned 


5.1  Technical  Lessons  Learned 

5.1.1  Interfaces 

Distributed  testing  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and 
simulations.  However,  carefiil  planning  can  significantly  reduce  the  potential  for  difficulties 
arising  from  network  interface  problems.  As  part  of  their  planning,  the  ETE  Test  team  bought 
standard  network  equipment  for  all  of  the  sites.  Thus,  the  configuration  of  the  ETE  Test 
environment  did  not  pose  any  problems  during  the  Phase  3  test. 

5.1.2  Instrumentation 

Special  equipment  was  necessary  for  ADS  network  check-out  and  verification.  Special  test 
equipment  and  networking  tools  will  rapidly  isolate  the  specific  cause  of  network  and  ADS/DIS 
problems.  Without  the  special  equipment,  troubleshooting  would  have  been  accomplished  by  trial 
and  error  increasing  time,  cost,  and  personnel.  In  addition,  the  key  N&E  personnel  should  be 
trained  in  the  use  of  the  special  test  equipment  and  networking  tools. 

5.2  Infrastructure  and  Process  Lessons  Learned 

5.2.1  Procedures 

5. 2.1.1  Planning 

The  requirements  for  an  ADS  test  must  be  clearly  defined  early  in  the  test  planning  phase. 
Detailed  planning  and  coordination  are  required  to  ensure  a  common  understanding  of  all 
requirements,  procedures,  and  test  objectives  since  individual  facilities  are  generally  unfamiliar 
with  conducting  coordinated,  distributed  T&E  tests.  Phase  3  testing  succeeded  because  of  close 
planning  and  coordination  among  the  ETE  Test  team  and  the  supporting  facilities  at  the  various 
nodes. 

5. 2. 1.2  Development 

Risk  reduction  testing  prior  to  actual  test  execution  will  help  test  team  personnel  identify  and 
resolve  potential  ADS  system  problems.  At  the  Northrop  Grumman  node,  extensive  laboratory 
testing  paved  the  way  for  the  successful  Phase  3  tests. 

5. 2. 1.3  Execution 

Briefings  are  needed  before  and  after  each  ADS  test.  These  briefings  should  include  such 
information  as  the  test  objectives,  telephone  numbers  to  use  for  test  control,  the  test  configuration 
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of  each  facility,  instrumentation  and  data  collection  requirements,  go/no  go  criteria,  contingency 
and  backup  plans,  and  test  conduct.  A  briefing  checklist  should  be  developed  and  used. 

5. 2. 1.4  Evaluation 

Effective  data  management  is  needed.  Linked  facilities  can  generate  a  large  volume  of  data  at 
distributed  locations.  Without  careful  planning,  key  data  may  not  be  collected  and/or  transmitted 
to  the  analysis  center,  and  data  collected  at  the  network  nodes  may  not  be  in  a  useful  form  for 
centralized  analysis.  Before  ADS  testing,  a  comprehensive  data  management  plan  must  clearly 
identify  the  data  to  be  collected  at  each  network  node,  onsite  processing  of  the  data,  and  the  data 
to  be  transmitted  to  the  analysis  center. 

5. 2. 1.5  Command  and  Control 

Have  test  controllers  who  are  extremely  familiar  with  the  test  and  network  configuration.  The 
test  controller  for  Phase  3  had  acted  as  test  controller  during  the  Phase  2  testing. 

Have  a  centralized  test  control  center.  The  JADS  TCAC  is  configured  to  allow  for  convenient, 
instant  communications  with  all  the  nodes.  It  acted  as  the  central  point  of  contact  between  the 
nodes  and  for  all  problems.  The  test  controller  kept  track  of  test  progress  and  documented  any 
problems  that  occurred. 

5.2.2  Policy 

Network  management  and  troubleshooting  must  be  disciplined  and  organized  with  a  thorough 
understanding  and  strong  configuration  control  of  the  ADS  network. 

5.2.3  Personnel 

Personnel  involved  in  a  distributed  test  should  understand  the  “big  picture.”  When  problems 
arise,  personnel  who  understand  the  entire  test  and  the  overall  network  will  find  solutions  much 
faster.  During  Phase  3,  the  ETE  Test  team  personnel  were  stationed  at  the  same  locations  as  they 
were  during  Phase  2  to  take  advantage  of  the  experience  gained  during  Phase  2. 
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6.0  Conclusions/Recommendations 


The  Phase  3  architecture  was  essentially  a  transition  between  the  Phase  2  and  Phase  4 
architectures  and  did  not  involve  any  C4ISR  DT&E  or  OT&E.  Thus  the  following  conclusions 
and  recommendations  are  based  on  cumulative  ETE  test  experience. 

6.1  Utility 

6.1.1  Utility  Conclusions 

6. 1.1.1  Enhanced  Testing.  An  ADS  environment  can  enhance  the  testing  of  C4ISR  systems. 
Compared  to  conventional  methods,  an  ADS  environment  can  realistically  test  C4ISR  systems 

•  with  larger  numbers  of  ground-based  entities  at  a  much  lower  cost. 

•  for  longer  periods  of  time,  enabling  increased  data  collection  and  the  ability  to  analyze  and 
improve  the  data  gathering  process. 

By  allowing  the  simulation  of  large  battlespaces  with  large  numbers  of  entities,  ADS  technology 
provides  testers  with  greatly  expanded  capabilities  for  test  concept  and  design. 

Testers  can  use  ADS  to  save  time,  resources,  and  test  personnel  man-hours  by  linking  several 
pieces  of  equipment  and/or  facilities  together  for  simultaneous  testing  instead  of  conducting 
individual  tests  at  different  locations. 

6.1.2  Utility  Recommendations 

Large  exercises  could  use  the  ETE  Test  environment  to  virtually  augment  the  battlefield  with 
simulated  targets.  During  Phase  4,  this  capability  will  be  demonstrated  with  the  integration  of  a 
live  E-8C  Joint  STARS  aircraft  into  the  ETE  Test  ADS  environment. 

An  ADS  environment,  like  the  ETE  Test  environment,  is  flexible  enough  to  allow  for  further 
expansion  and  increased  opportunities  for  testing  C4ISR  systems.  The  Janus  battlespace  can  be 
expanded  as  required.  Increasing  the  number  of  LGSMs  or  CGSs  would  create  more  realistic 
targeting  capabilities.  By  adding  other  assets  to  the  environment,  such  as  an  unmanned  aerial 
vehicle  (UAV)  or  a  tactical  aircraft  simulator,  the  robustness  of  the  environment  could  be 
significantly  enhanced. 

6.2  Technical 

6.2.1  Technical  Conclusions 

The  ETE  Test  network  was  highly  reliable  during  Phase  3  testing. 
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As  expected,  the  Phase  3  testing  at  the  Northrop  Grumman  node  showed  that  all  of  the  available 
satellite  link  bandwidth  was  required  for  data  transmission,  and  that  buffering  was  needed  at  times 
to  handle  periods  of  heavy  scenario  activity.  Without  buffering,  the  satellite  link  exhibited  a 
normal  latency  of  around  two  seconds.  With  buffering,  the  latency  approached  six  seconds. 
Neither  of  these  latencies  was  observable  in  the  radar  reports,  indicating  that  the  ETE  Test 
synthetic  environment  is  very  tolerant  of  latencies  in  this  range.  However,  ADS  test  planners 
need  to  consider  these  factors  when  testing  other  C4ISR  systems  involving  satellite  links. 

6.2.2  Technical  Recommendations 

With  careful  planning  and  resource  management,  testers  can  address  the  issues  associated  with 
integrating  simulations  into  an  ADS  test  environment. 

•  Identify  the  assumptions  and  limitations  associated  with  those  simulations. 

•  Budget,  schedule,  and  provide  the  manpower  necessary  to  develop  the  simulations. 
Simulation  development  is  typically  labor  intensive  and  thus  costly. 

•  Determine  the  level  of  simulation  detail  needed  for  the  ADS  test.  Development  costs  are 
directly  related  to  the  level  of  simulation  detail. 

•  Identify  and  provide  training  for  the  users  of  the  simulations. 

6.3  Infrastructure 

6.3.1  Infrastructure  Conclusions 

ADS  can  reduce  the  number  of  troops  and  associated  equipment  involved  in  tests  because  of  its 
simulation  of  fielded  forces.  However,  the  ADS  infrastructure  requires  technical  personnel  to  set 
up  and  execute  the  tests  and  to  analyze  the  test  results. 

Highly  structured  test  control  is  a  key  ingredient  for  ADS  test  success.  This  test  control  should 
include  formalized  procedures  with  an  emphasis  on  checklists. 

An  ADS  test  can’t  always  count  on  having  the  personnel  requirements  for  a  distant  node  supplied 
by  an  organization  local  to  the  node.  Even  if  an  ADS  test  is  able  to  employ  these  people,  it  may 
then  lose  them  to  other  activities  deemed  more  important  by  the  local  organization.  During  Phase 
3,  the  ETE  Test  team  deployed  two  of  its  most  experienced  members  to  the  Northrop  Grumman 
node  to  ensure  effective  communication  and  coordination  with  the  activities  occurring  at  this 
node. 

An  ADS  environment  necessitates  sophisticated  instrumentation  with  rigorous  processing  speed, 
data  storage,  and  data  integration  capabilities.  This  instrumentation  can  be  costly  and  can  require 
trained  personnel  for  its  successful  operation. 
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ADS  analysts  must  have  a  well-planned  and  organized  approach  to  managing  the  large  amounts  of 
data  produced  from  ADS  testing. 

6.3.2  Infrastructure  Recommendations 

Make  every  effort  to  simplify  the  infrastructure.  Time  spent  in  the  planning  stages  of  an  ADS 
test,  with  an  emphasis  on  reducing  the  complexity  of  the  test  network,  is  time  well  spent.  Use 
proven  hardware  and  keep  it  the  same  wherever  possible. 
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APPENDIX  A  -  JADS  Test  Procedures 


A1.0  Test  Procedures 

Various  types  of  checklists  were  used  during  the  execution  of  the  Phase  3  test.  The  Test  Control 
and  Analysis  Center  (TCAC)  test  controller  checklist  can  be  found  in  Section  Al.l,  TCAC  Test 
Procedures.  This  checklist  was  used  to  ensure  network  and  logger  functionality  and  to  provide 
overall  test  control  procedures.  Each  node  (White  Sands  Missile  Range  [WSMR],  Northrop 
Grumman,  and  Fort  Sill)  incorporated  the  logger  functions  from  the  TCAC  checklist  into  their 
own  checklist. 

Other  checklists  were  used  to  direct  the  operation  of  various  pieces  of  test  equipment.  An 
example  is  included  in  Section  A1.2,  TCAC  Plan  View  Display  (PVD)  Procedures. 

Section  A1.3,  WSMR  Procedures,  is  representative  of  the  site-specific  checklists.  WSMR, 
Northrop  Grumman  and  Fort  Sill  all  developed  procedures  for  operation  of  the  End-to-End 
(ETE)  Test  environment  equipment.  Only  Fort  Hood,  the  only  site  without  a  logger,  failed  to 
develop  written  procedures.  Their  procedures  were  primarily  accomplished  by  resident  specialists 
having  their  own  procedures. 

Al.l  TCAC  Test  Procedures 

The  following  are  the  written  test  procedures  used  in  the  TCAC  during  Phase  3  testing. 

72  HOURS  PRIOR  TO  TEST 

Network  Coordinator: _ 

Date: _ Test  Time: _ to _ 

1.  _ Check  supplies. 

2.  Turn  on  equipment. 

_  a.  Turn  on  3  Barcos  (Spectrum  [Sun5]  on  1,  Janus  [hp735]  on  2,  and  NetVis  [indigo2]  on  3). 

b.  Log  in  as  “root”  to  indigo2  in  the  TCAC,  and  indy4  in  communications  room  1 . 

_  1)  From  the  toolchest,  select  Toolbox,  JADS  Toolbox,  Monitor,  PDU  Monitor,  PDU 

Statistics,  Show  Stats  to  display  protocol  data  units  (PDUs). 

_  2)  From  the  toolchest,  select  NetVis,  NetGraph-ETE  to  display  network  traffic. 

_  3)  From  the  toolchest,  select  NetTests,  Status  check  ETE  to  start  and  display  network 

connectivity  tests,  (uts  in  comm  rm  1  pings  wsmr,  ftsill,  and  fthoodafatads.  indigo2, 
pings,  gnimman,  indy3,  and  sparc5  at  Ft  Hood). 

_ c.  In  the  TCAC,  run  Spectrum  on  the  Sun20  (server)  and  Sun5  (graph)  to  display  Zulu  time  and 

router  status. 

_  d.  Create  an  empty  file  “touch  /scripts/ .go”  in grumman,  indigo2,  and indy4. 
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Clear  router  interfaces.  To  clear  the  grumman_router,  jads_router,  and  fthoodjrouter  from 
indigo2;  and  fthoodjrouter,  ftsill_router,  and  wsmr_router  from  indy4,  run: 
“/scripts/clear_router  ete.” 

Not  used. 

Time  accuracy.  Verily  that  each  site  has  network  time  protocol  (NTP)  running. 

a.  From  uts,  run  “/scripts/check_time”  and  verify  that  the  offsets  for  ftsill  and  wsmr 
are  less  than  1  millisecond  (ms). 

b.  From  indigo2,  rlogin  to  indyl,  run  “/scripts/check_time .  ”  Verify  offsets  for 
grumman,  indy3,  and  sparcS  are  less  than  1  ms. 

Available  disk  space.  Verify  that  each  logger  has  at  least  600  megabytes  (MB)  of  unused  disk 
space  available  on  the  /disk2  partition. 

a.  From  uts,  rlogin  to  ftsill  and  wsmr,  in  turn,  and 
from  indigo2,  rlogin  to  grumman,  and  indy3,  in  turn. 

b.  Run  “df  -  k”  on  each  machine  (including  uts)  to  display  the  available  disk  space.  Verify  that 
each  has  at  least  600  MB  available. 

Port  settings.  Verify  that  each  logger  is  set  to  port  3000  and  the  exercise  identification  (ID)  is  0. 

a.  From  uts,  rlogin  to  ftsill  and  wsmr,  in  turn,  and 
from  indigo2,  rlogin  to  grumman,  and  indy3,  in  turn. 

b.  Run  “more  /scripts/dt_logger”  to  view  the  file.  Look  for  the  entry: 

“  /usr/local/bin/jads_logger  3000  0  /disk2/logf iles 

/$testdate"_test"$testnum"_"$runnum/'_''$site#/  •  log"  ”  entry  in 
two  places. 

Voice  conference  net.  Verify  the  net  is  functional  by  dialing  in  from  two  different  phones  in  the 
TCAC  at  the  same  time  to  establish  the  net. 

Not  used. 


Data  collection  test  a: 

a.  From  uts,  rlogin  to  ftsill  and  wsmr,  simultaneously,  and 
from  indigo2,  rlogin  to  grumman,  and  indy3,  simultaneously. 

b.  Start  the  ftsill,  grumman,  indy3,and  uts  loggers  using  test  number  “0  00”  and  run  number 
“a”  (i.e.  -  “/scripts/dt_logger  000  a”). 

c.  Run  the  “/scripts/run_player  3000  /disk2/logf iles/ne_test . log” 
file  on  the  wsmr  machine. 

d.  Determine  when  run  is  complete.  Stop  all  loggers  (“C  tr  1  -  C”). 

e.  Check  digital  communications  terminal  (DCT)  results.  Verify  reception  of  2281  PDUs  on 
grumman,  indy3,  and  uts  (or  indy4)  loggers.  (No  PDUs  at  ftsill). 

Data  collection  test  b: 

a.  From  uts,  rlogin  to  ftsill  and  wsmr,  simultaneously,  and 
from  indigo2,  rlogin  to  grumman,  and  indy3,  simultaneously. 

b.  Start  the  grumman,  indy3,  uts  and  wsmr  loggers  using  test  number  “0  00”  and  run  number 
“a”  (i.e.  -  “/scripts/dt_logger  000  a”). 
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_ _  c.  Runthe“/scripts/run_player  3000  /disk2/logfiles/ne_test • log” 

file  on  the  ftsill  machine. 

_  d.  Determine  when  run  is  complete.  Stop  all  loggers  (“Ctrl  -  C”). 

_ e.  Check  DCT  results.  Verify  reception  of  2281  PDUs  on  grumman,  indy3,  and  uts  loggers. 

12.  _ Report  the  results  of  the  network  checks  to  the  test  controller.  Supervise  repairs  as  necessary  to 

prepare  equipment  for  the  test  sequence. 


PRETEST  (DAY  OF  TEST) 

Network  Coordinator: _ 

Date: _ Test  Time: _ to _ 

1 .  _ _ Check  supplies.  Provide  checklists,  blank  log  sheets,  file  name  lists,  pens,  pencils,  scratch  paper, 

and  4  millimeter  (mm)  tape  cartridges  for  the  test. 

2.  Turn  on  equipment. 

_  a.  Turn  on  3  Barcos  (Spectrum  [Sun5]  on  1,  Janus  [hp735]  on  2,  and  NetVis  [indigo2]  on  3). 

b.  Log  in  as  “root”  to  indigo2  in  the  TCAC,  and  indy4  in  communications  room  1. 

_  1)  From  the  toolchest,  select  Toolbox,  JADS  Toolbox,  Monitor,  PDU  Monitor,  PDU 

Statistics,  Show  Stats  to  display  PDUs. 

_  2)  From  the  toolchest,  select  NetVis,  NetGraph-ETE  to  display  network  traffic. 

_  3)  From  the  toolchest,  select  NetTests,  Status  Check  ETE  to  start  and  display  network 

connectivity  tests,  (uts  in  Comm  Rm  1  pings  wsmr,  ftsill,  and  fthoodafatads.  indigo2, 
pings,  grumman,  indy3,  and  sparc5  at  Ft  Hood). 

_  c.  In  the  TCAC,  run  Spectrum  on  the  Sun20  (server)  and  SunS  (graph)  to  display  Zulu  time  and 

router  status. 

3.  Clear  router  interfaces.  To  clear  the  grammanjrouter,  jadsjrouter,  and  fthood_router  from 
indigo2;  and  fthood__router,  ftsilLrouter,  and  wsmr_router  from  indy4,  run: 

_  “/scripts/clear_router  ete.” 

4.  _ Not  used. 

5.  Time  accuracy.  Verify  that  each  site  has  NTP  running. 

_  a.  From  uts,  run  “/scripts/check_time”  and  verify  that  the  offsets  for  ftsill  and  wsmr 

are  less  than  1  ms. 

_ b.  From  indigo2,  rlogin  to  indyl,  run  “/scripts/check_time .  ”  Verify  offsets  for 

grumman,  indy3,  and  sparc5  are  less  than  1  ms. 

6.  _ Available  disk  space.  Performed  at  each  logger  by  the  logger  operator. 

7.  _ Port  settings.  Performed  at  each  logger  by  the  logger  operator. 
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8.  _ Join  the  voice  conference  net.  Both  the  test  controller  and  the  network  coordinator  (NC)  dial 

61143  in  the  TCAC  to  establish  the  conference  net. 

9.  _  Time  synchronization,  ftsill,  grumman,  indy3,  and  wsmr  operators  check  global  positioning 

system  (GPS)  time  reception  by  typing  “date”  and  press  Enter  on  the  NC’s  mark.  Report  time  to 
NC. 

(NOTE:  indyl  is  time  server  for  classified,  uts  is  time  server  for  unclassified.) 

10.  Data  collection  test  a: 

_ a.  Cue  ftsill,  grumman,  indy3,and  uts  operators  to  start  loggers  using  test  number  “0  00”  and 

run  number  “a”  (i.e.  -  “/scripts/dt_logger  000  a”). 

_ b.  Cue  wsmr  operator  to  run  “/scripts/run_player  3000 

/disk2/logf  iles/ne_test .  log”  file. 

_ c.  Determine  when  run  is  complete.  Cue  all  operators  to  stop  loggers  (“Ctrl  -  C”). 

_  d.  Check  DCT  results  -  Have  grumman,  indy3,  and  uts  operators  verify  reception  of  2281 

PDUs.  (No  PDUs  at  ftsill). 

11.  Data  collection  test  b: 

_ a.  Cue  grumman,  indy3,  uts,  and  wsmr  operators  to  start  loggers  using  test  number  “000”  and 

runnumber  “b”  (i.e.  -  “/scripts/dt_logger  000  b”). 

_  b.  Cue  ftsill  operator  to  run  “/scripts/run__player  3000 

/disk2/logf  iles/ne_test .  log”  file. 

_  c.  Determine  when  run  is  complete.  Cue  all  operators  to  stop  loggers  (“Ctrl  -  C”). 

_  d.  Check  DCT  results  -  Have  grumman,  indy3,  uts,  and  wsmr  operators  verify  reception  of 

2281  PDUs.  (No  PDUs  at  ftsill). 

12.  _  Report  the  results  of  the  network  checks  (items  9-1 1)  to  the  test  controller. 


The  pretest  phase  is  now  complete.  Proceed  to  the  test  run  phase. 

NOTE:  Sometimes  the  logger  process  does  not  terminate  on  the  grumman  logger.  In  that  case,  run 
/scripts/f  ind_logger  on  the  grumman  logger  to  kill  the  process  and  delete  the  old  logfile  before 
restarting  the  logger  with  the  same  filename. 


TEST  RUN 

Network  Coordinator: _ 

Date: _  Lab  Time: _ to _ 

1 .  _  Start  loggers.  Obtain  the  test  and  run  numbers  from  the  test  controller  and  record  on  the  log 

sheet.  Operators  are  cued  by  the  test  controller  when  to  start  loggers.  Record  start  time  on  the 
log  sheet. 

_  a.  Early  in  the  test  run,  verify  with  operators  that  all  loggers  are  receiving  PDUs  (number  is 

increasing. 

_ b.  Periodically  check  with  operators  that  all  loggers  continue  to  receive  PDUs  (number  is 

increasing). 
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_  c.  Periodically  check  “bat”  phone  operation  if  not  used  regularly. 

_ d.  Every  Vi  hour,  run  a  time  accuracy  check.  From  uts,  run  “/scripts/check_time”  to 

check  ftsill  and  wsmr.  From  indigo2,  rlogin  to  indyl  and  run  “/scripts/check_time”  to 
check  indyl,  grumman,  and  tcacindy.  Time  offsets  should  be  <1  ms. 

_ e.  Keep  written  event  log. 

2.  Stop  loggers.  Loggers  stop  recording  data  when  directed  by  the  test  controller  (“Ctrl-C”). 

_ a.  Record  the  stop  time  and  the  total  number  of  PDUs  from  each  logger  on  log  sheet. 

_ b.  Confirm  that  the  required  data  have  been  logged.  From  uts,  rlogin  to  ftsill  and  wsmr  and 

run  “Is  -1  /disk2/logf iles .”  Fromindigo2,  rlogin  to  indy3,  and  grumman  and 
run  “Is  -1  /disk2/logf  iles.”  Check  the  file  sizes;  the  filename  is 
“ mmddyy_test#-run#_loggernameAog .” 

3.  _  Subsequent  runs.  When  additional  runs  are  required,  repeat  steps  1  and  2  for  each  run. 


The  test  run  phase  is  now  complete.  Proceed  to  the  post-test  phase. 


POST  TEST  (DAY  OF  TEST). 

Network  Coordinator: _ 

Date: _  Test  Time: _ to _ 

1 .  Remote  file  capture.  Consolidate,  compress,  and  copy  the  test  run  logger  files  from  each  remote 
site. 

_ __  a.  For  classified  data,  rlogin  to  each  logger  (grumman  and  indy3),  in  turn,  from  tcacindy  in  the 

TCAC,  or 

For  unclassified  data,  rlogin  to  each  logger  (ftsill,  wsmr,  and  uts),  in  turn,  from  uts. 

_ b.  If  only  1  file  for  the  day  exists  in  the  logger  at  a  site,  skip  to  step  c.  If  more  than  1  log  file  for 

the  day  exists  at  a  site,  consolidate  them  by  using  the  command 

“tar  cvf  mmddyy_si tename. log. tar  mmddyy* . log  ” 

where  *  is  the  wildcard  character  that  includes  all  the  files  for  that  day  for  that  site  name. 

(e.g.,  -  “tar  cvf  040798_wsmr .  log.  tar  040798*  .  log”  ). 

_ c.  Compress  the  single  log  file  (e.g.,  -  “compress  04079 8_wsmr .  log”)  or  the  tar  file 

from  step  b  (“compress  0407  98_wsmr .  tar”). 

_ d.  On  tcacindy,  run  “/scripts/rcp  etef  ile”  to  copy  the  tar'd  and  compressed  classified 

files  (ummddyy__sitename. log.Z”)  from  both  grumman  and  indy3  loggers  to 
tcacindy:/disk 2/etelmmddyyL 

_ e.  On  uts,  run  “/script  s/rcp_e  tef  ile”  to  copy  the  tar’d  and  compressed  unclassified 

files  (“mmddyy_sitename. log.z”)  from  ftsill,  uts,  and  wsmr  loggers  to 
_  uts:/di$\s2/etelmmddyy/ . 

_ f.  Copy  the  unclassified  files  from  step  e  to  4mm  tape  and  activate  the  write  protect  feature. 

g.  Move  the  tape  to  tcacindy  and  copy  the  unclassified  files  from  the  tape  to  /disk2/et d mmddyy! 
(i.e.,  from  the  ete  directory,  run  tar  xv  to  extract  the  files  from  the  tape  to  the  hard  drive). 

Make  sure  the  write  protect  feature  is  ON. 

2.  Backup  tapes. 
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_ a.  Create  a  backup  tape  of  the  files  in  tcacindy:/disk2/ete//H//uWjy/  using  either  the  “  tar  cv 

mmddyy  ”  command  while  in  the  ete  directory  (or  the  tape  tool  on  the  tcacindy  desktop), 
b.  Verify  the  backup  using  either  the  “tar  tv”  command  or  the  tape  tool. 

_ c.  Remove  the  tape  from  the  drive  and  label  it. 

_ d.  Repeat  a,  b,  and  c  to  create  a  duplicate  tape. 

_ e.  Deliver  both  tapes  to  the  ETE  Test  team  representative. 

3.  _ Delete  the  data  collection  test  and  the  backed-up  log  files  from  /disk2/Iogfiles/  on  all  loggers. 

4.  _ On  the  last  day  of  testing,  delete  the  file  “/scripts/ .  go”  in  grumman,  indigo2,  and  indy4. 

5.  _ Logoff  from  logger.  Turn  Off  the  monitor,  but  leave  the  central  processing  unit  (CPU)  On! ! ! 

6.  _ Participate  in  mission  debrief,  if  applicable. 


A1.2  TCAC  Plan  View  Display  (PVD)  Procedures 

The  following  procedures  were  used  to  initiate  test  monitoring  with  the  Janus  plan  view  display 
program.  This  is  representative  of  the  specific  checklists  developed  to  aid  in  the  operation  of  test 
equipment. 


Functionality/Integration  Test  Checklist 
(TCAC-PVD) 


Date: 

Scenario: 


Test  Stop  Time  (z): 
Scenario  Stop  Time: 


Test  Start  Time  (z): 
Scenario  Start  Time: 


Step# 

POC 

Action 

Event 

Go/No 

Go 

Run  PVD  ! 

1 

ETE 

Power  on  the  hp735  monitor. 
Log  on  to  the  hp735  as  hovey. 

2 

ETE 

From  the  xterm  window  that 
appears,  type  pvd,  and  hit  enter. 

Use  this  alias  to  start 

Janus  plan  view  display. 

3 

ETE 

From  the  Janus  plan  view 
display  menu,  verify  the 
parameters  for  the  run: 
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Workstation  1 

Terrain  File _ 

Screen  File _ 

Symbol  File  3 
Symbol  Size  10 
Terrain  File  Meridian  45 
Exercise  ID  BLANK 
Map  Spheroid  1 
Mode  BLANK 
Terminate  this  Run  N 

and  hit  keypad  enter. 


Wait  until  the  PVD  terrain  and 
combat  systems  databases  are 
loaded. 

Double  click  the 
Analyst_Workstation_WSl  icon 
to  bring  up  the  scenario 

window. _ 

From  the 

Analyst_Workstation_WSl 
scenario  window,  functions 
menu, 

left  click  Draw  CAC. 


Use  the  correct  terrain  file 
and  screen  file  for  the 
vignette. 


Last  message:  Opening 
file 

.  Jjads_ete/trn/TSCRN_ 
DAT 


7 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window,  CAC  File 
menu,  select  the  CAC  file 
number  to  display. 

Left  click  increases  number,  and 
right  click  decreases  number. 

Left  click  Add  to  display  the 
CAC. 

Places  command  and 
control  overlays  on  the 
scenario  box. 

8 

ETE 

From  the 

Analyst_  Workstation_  WS1 
scenario  window,  function 
menu, 

left  click  Display. 

Ready  to  receive  and 
display  DIS  PDUs. 

9 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window  menu: 

Left  click  any  tick  on  the  zoom 
in/out  menu,  then  select  the 

Used  as  necessary  to 
zoom  in/out  of  the 
scenario  box. 
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Step# 

POC 

Action 

Event 

Go/No 

Go 

StoD  PVD 

1 

ETE 

From  the 

Analyst_Workstation_WSl 
scenario  window, 
right  click  End. 

Shuts  down  PVD. 

2 

ETE 

Minimize  the 

Analyst _  Workstation_  WS1 
scenario  window. 

In  the  xterm  that  remains, 
verify  this  message: 

STOP  - — JANPVD 
Program  Terminated - 

3 

ETE 

From  the 

Analyst_Workstation_WSl 

icon, 

right  click  and  choose  close. 

Closes  the  scenario 
window. 

Step  # 

POC 

Action 

Event 

Go/No 

Go 

|  Shut  Down  Test  1 

i 

ETE 

Left  click  EXIT  from  the  HP 
VUE  front  panel. 

Signs  off  the  hp735. 
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2 

ETE 

Left  click  Continue  logout  from 
the  dialog  box. 

Confirms  desire  to  log 
out. 

3 

ETE 

Power  off  the  hp735  monitor. 

A1.3  WSMR  Procedures 

This  checklist  is  representative  of  the  individual  site  checklists.  It  incorporates  the  logger 
functionality  and  the  site  specific  actions  required  by  the  operators).  These  are  maintained  by  the 
site  specialists  and  updated  as  changes  are  required. 

Functionality/Integration  Test  Checklist 
(WSMR) 


Date:  _ 

Scenario:  _ 

Janus  File:  _ 

Indy  File:  _ 

Test  Start  Time  (z): 
Scenario  Start  Time: 


Test  Stop  Time  (z): 
Scenario  Stop  Time: 


Step  # 

POC 

Action 

Event 

Go/No 

Go 

|  Network  Activation 

i 

Verify  operation  of  hotlink 

Enables  secure  and 

phone.  If  no  go,  contact  N&E 

unclassified  voice 

to  fix  the  network. 

communications. 

2 

ETE 

Verify  that  WSMR  indy  and  the 

Initial  step  in  ensuring 

and 

WSMR  hp715  are  on  the  JADS 

network  is  operational. 

WSM 

R 

ETE  network. 

3 

N&E 

Verity  N&E  has  cleared  and 

Clears  router  interface 

reset  routers. 

cards. 

4 

ETE 

Power  on  the  WSMR  monitor. 
Log  on  to  WSMR  as  dislog. 
From  a  Unix  shell  window  as 
su,  run  /scripts/restart. 

Restarts  the  indy. 

5 

ETE 

After  a  successful  restart,  log 

Signon  is  used  for 

on  to  wsmr  as  dislog. 

checking  network 
communications  and 

logging  PDU  data. 
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ETE 

From  a  Unix  shell  window  as 

Verifies  that  each  network 

su, 

link  is  operational.  3% 

run  /scripts/pingjest  to  get 

loss  at  Fort  Sill  and  uts  is 

ping  statistics  for  each  remote 
site. 

normal. 

58 


7 

ETE 

From  a  Unix  shell  window  as 
su,  run  /scripts/check  Jime. 

Displays  the  offset  from 
uts.  Should  be  less  than  1 

ms. 

ETE 

From  a  Unix  shell  window  as  su 
and  at  the  test  controller’s 
direction, 

run  /scripts/run _player  3000 
/disk2/logfiles/ne_test.log  to 
check  ability  to  send  PDUs  to 
each  remote  site. 

Verifies  sending  2281 

PDUs  and  receiving  the 
same  number  of  PDUs  at 
each  remote  site. 

9 

ETE 

From  a  Unix  shell  window  as  su 
and  at  the  test  controller’s 
direction, 

run  / scripts/ dt  logger 

to  check  ability  to 
receive  PDUs  from  a  remote 
site.  At  the  test  controller’s 
direction,  hit  Ctrl-C  to  end  the 
logfile. 

Verifies  receiving  2281 
PDUs  from  a  remote  site. 

10 

ETE 

From  a  Unix  shell  window, 
cd  /usr /local/bin  and 
run  ./display _pdu_rate. 

Select  port  3000  0. 

Left  click  start. 

Verifies  that  PDU_rate  = 

0.  Ensures  that  there 
aren’t  any  DIS 
communications  before 
the  start  of  testing. 
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Step  # 

POC 

Action 

Event 

Go/No 

Go 

Start  WSMR  Logger 

1 

ETE 

From  a  Unix  shell  window  as  su 
on  WSMR,  run 
/scripts/dt  logger 

Script  that  runs  the  JADS 
logger. 

2 

ETE 

Verify  the  logfile  name  as 
/disk2/logfiles/  vus 

mr.log  and  port  3000. 

Opens  port  3000  to  listen 
and  log  all  DIS 
communications.  Writes 
to  the  listed  logfile. 

Step# 

POC 

Action 

Event 

Go/No 

Go 

Start  Janus 

ETE  Power  on  the  c  1 80  monitor. 

Log  on  to  the  cl 80  as  JADS. 

ETE  From  an  hpterm  window,  type 
janus.e xe,  and  hit  enter. 

ETE  Left  click  PE  (Program 

Execution)  from  the  Janus  User 

_ Options  menu. _ 

ETE  Left  click  JE  (Janus  Execution) 
from  the  Program  Execution 

_ menu. _ 

ETE  Type  desired  scenario  number 

_ for  the  run,  and  hit 

enter. 

Type  run  number  1,  and  hit 
enter. 


ETE  Hit  enter  again  to  continue. 


Use  this  executable  to 
start  Janus. 


Brings  up  the  Program 
Execution  menu. 


First  step  in  defining  the 
scenario. 


Tells  Janus  which  scenario 
to  run. 


Ready  to  continue. 


ETE  Verify  that  1  is  entered.  Hit  Use  a  normal  run. 
enter  one  more  time. 


ETE  From  the  Janus  Runtime 
Screens  menu, 
left  click  11. 

Verify  time  of  day  is  correct  for 
the  vignette,  and  hit  keypad 
enter. 


Verifies  time  of  day. 
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ETE  From  the  Janus  Runtime 
Screens  menu, 
left  click  22. 

Verify  that  there  is  a  setup  for: 
WS Number  7, and  Side  1, 
and  hit  keypad  enter.  _ 


ETE  From  the  Janus  Runtime 
Screens  menu, 
left  click  66. 

Verify  the  DIS  operational 
parameters  for  the  run: 

Janus  side  1 
DIS  side  2 
DIS  COMM  calls/sec 

Units  processed/COMM  call 

Terrain  File  Meridian  (+E)  45 
Heartbeat(s) 


Verifies  that  a  controller 
workstation  has  been 
configured. 


Verifies  DIS  parameters. 
Calculate  the  new 
heartbeat  as  follows: 

C  x Rx  H  <  T 

where 

C  =  calls/sec, 

R  =  units/call, 

H  =  heartbeat,  and 
T  =  total  number  of  units 
in  scenario 


Dead  Reckoning  Threshold 

999 

Site  TRAC-WSMR  23 
Host  CPU  HP  4 
Exercise  JADS-ETE  4 
DIS  version  transmit  4 
DIS  version  receive  4 


ETE 

Left  click  JJ  (Begin  Janus) 
from  the  Janus  Runtime  Screens 

menu. 

Loads  the  Janus  scenario. 

ETE 

Wait  until  the  Janus  scenario 
loads.  Verify: 

Scenario  number 

Total  number  of  units 

ETE 

Double  click  the  sidel  icon  to 
bring  up  the  scenario  window. 

This  brings  up  the 
scenario  window  which 
allows  a  Janus  operator  to 
interact  (game)  the 
exercise. 

Step  # 

POC 

Action 

Event 

Go/No 

Go 

Run  Scenario 

1 

ETE 

From  the  sidel  scenario 

window, 

left  click  DIS. 

2 

ETE 

From  the  sidel  scenario 
window, 

left  click  START. 

3 

ETE 

Minimize  the  Janus  scenario 
window  {sidel).  Type  rr  in  the 
Janus  window,  and  hit  enter. 

4 

ETE 

Type  n  and  hit  enter. 

5 

ETE 

Hit  enter  again. 

6 

ETE 

Double  click  the  Janus  scenario 
window  {sidel). 

7 

ETE 

Verify  that  loggers  are  logging. 

DIS  button  highlights. 
Opens  DIS 
communications. 


First  step  in  running  a 
Janus  scenario. 


Ready  to  continue  the 
Janus  run. 


No  planned  save. 


Default  checkpoint 
frequency. 


Verifies  scenario 
movements  and  a  running 
time  of  day  counter. 


Step  # 

POC 

Action 

Event 

Go/No 

Go 

Stop  Scenario 

1 

ETE 

From  the  sidel  scenario 

window, 

left  click  DIS. 

DIS  button  unhighlights. 
Closes  DIS 
communications. 

2 

ETE 

From  the  sidel  scenario 
window, 

right  click  ADMIN. 

Brings  up  options  menu. 

3 

ETE 

Left  click  EJ  (End  Janus). 

Quits  the  scenario  run. 

4 

ETE 

Right  click  2  times. 

Completely  closes  Janus. 

5 

ETE 

Left  click  EXIT  from  the  HP 
VUE  front  panel. 

Sign  off  the  hp715. 
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Step  # 

POC 

Action 

Event 

Go/No 
-  Go 

Shut  Down  Test 

Power  off  the  cl  80  monitor, 
and  shutdown  CPU. 

Make  sure  that  JADS  N&E 

FTP 

/disk2/lo<3 files/  ws 

mr.log 

back  to  JADS  and  place  the  file 
in 

/usr/testdata2/logs/ete/DDMM 

YY 

Ensures  data  integrity. 

This  file  will  be  analyzed 
by  JADS  analysts. 

Power  off  the  wsmr  monitor. 

After-action  review 

63 


APPENDIX  B 

Joint  STARS  Multiservice  Operational  Test  and  Evaluation 
Phase  3  Measures  Correlation 


May  1999 


DISTRIBUTION  E.  Distribution  authorized  to  Department  of  Defense  components  only, 
critical  technology,  as  determined  on  15  May  1999.  Requests  for  this  document  made  before  1 
March  2000  shall  be  referred  to  JADS  JTF,  2050A  2nd  Street  SE,  Kirtland  Air  Force  Base,  New 
Mexico,  87117-5522.  After  1  March  2000,  requests  shall  be  referred  to  HQ  AFOTEC/HO,  8500 
Gibson  Blvd  SE,  Kirtland  Air  Force  Base,  New  Mexico  871 17-5558,  or  SAIC  Technical  Library, 
2001  North  Beauregard  St.  Suite  800,  Alexandria,  Virginia  223 11. 

WARNING  -  This  document  contains  technical  data  whose  export  is  restricted  by  the  Arms 
Export  Control  Act  (Title  22,  U.S.C.,  Section  275 1 ,  et  seq.)  or  Executive  Order  12470.  Violators 
of  these  export  laws  are  subject  to  severe  criminal  penalties. 

DESTRUCTION  NOTICE  -  Unclassified,  limited  distribution  documents,  destroy  by  any 
method  that  will  prevent  disclosure  of  contents  or  reconstruction  of  the  document. 


64 


65 


Appendix  B 
Pages  65-72 
intentionally  removed 


APPENDIX  C  -  Glossary 


A 

Accreditation.  See:  distributed  simulation  accreditation,  model/simulation  accreditation. 

Accuracy.  The  degree  of  exactness  of  a  model  or  simulation  relative  to  an  established  standard 
with  high  accuracy  implying  low  error.  [DIS] 

Activity.  An  event  that  consumes  time  and  resources  and  whose  performance  is  necessary  for  a 
system  to  move  from  one  event  to  the  next.  [DIS] 

Advanced  Distributed  Simulation  (ADS).  A  set  of  disparate  models  or  simulations  operating  in 
a  common  synthetic  environment.  The  ADS  may  be  composed  of  three  modes  of  simulation: 
live,  virtual  and  constructive,  where  the  latter  can  be  seamlessly  integrated  within  a  single 
exercise.  See  also:  live  simulation;  virtual  simulation;  constructive  simulation.  [DIS] 

Aggregate.  An  activity  that  combines  individual  entities  into  a  singular  entity.  Contrast  with: 
disaggregate.  [DIS] 

B 

Battlespace.  The  three-dimensional  battlefield.  [DIS] 

Benchmark,  (v)  The  activity  of  comparing  the  results  of  a  model  or  simulation  with  an  accepted 
representation  of  the  process  being  modeled,  (n)  The  accepted  representation  of  the  modeled 
process.  [DIS] 

Bit.  The  smallest  unit  of  information  in  the  binary  system  of  notation.  [IEEE  1278.1] 

Broadcast.  A  transmission  mode  in  which  a  single  message  is  sent  to  all  network  destinations, 
i.e.,  one-to-all.  Broadcast  is  a  special  case  of  multicast.  Contrast  with:  multicast;  unicast. 
[IEEE  1278.2] 

c 

Compatible.  Two  or  more  simulations  are  distributed  interactive  simulation  (DIS)  compatible  if 
(1)  they  are  DIS  compliant,  and  (2)  their  models  and  data  that  send  and  interpret  protocol 
data  units  (PDUs)  support  the  realization  of  a  common  operational  environment  among  the 
systems  (coherent  in  time  and  space).  Contrast  with:  compliant,  interoperable.  [DIS] 

Compliant.  A  simulation  is  distributed  interactive  simulation  (DIS)  compliant  if  it  can  send  or 
receive  protocol  data  units  (PDUs)  in  accordance  with  the  Institute  of  Electrical  and 
Electronics  Engineers  (IEEE)  Standard  1278  and  1278  (working  drafts).  A  specific  statement 
must  be  made  regarding  the  qualifications  of  each  PDU.  Contrast  with:  compatible, 
interoperable.  [DIS] 

Conceptual  Model.  A  description  of  the  content  and  internal  representations  which  are  the  user's 
and  developer's  combined  concepts  of  the  exercise.  It  includes  logic  and  algorithms  and 
explicitly  recognizes  assumptions  and  limitations.  [DIS] 

Constructive  Simulation.  Models  and  simulations  that  involve  simulated  people  operating 
simulated  systems.  See  Also:  war  games;  higher  order  model  (HOM).  [DIS] 
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Continuous  Model.  (1)  A  mathematical  or  computational  model  whose  output  variables  change 
in  a  continuous  manner;  that  is,  in  changing  from  one  value  to  another,  a  variable  can  take  on 
all  intermediate  values.  For  example,  a  model  depicting  the  rate  of  air  flow  over  an  airplane 
wing.  Syn:  continuous-variable  model.  (2)  A  model  of  a  system  that  behaves  in  a  continuous 
manner.  Contrast  with:  discrete  model.  [DIS] 

Continuous  Simulation.  A  simulation  that  uses  a  continuous  model.  [DIS] 
Continuous-Variable  Model.  See:  continuous  model.  [DIS] 

Control  Station.  (1)  A  facility  which  provides  the  individual  responsible  for  controlling  the 
simulation  and  the  capability  to  implement  simulation  control  as  protocol  data  units  (PDUs) 
on  the  distributed  interactive  simulation  (DIS)  network. 

Syn:  simulation  -  management  station.  [DIS] 

D 

Data.  Representation  of  facts,  concepts,  or  instructions  in  a  formalized  manner  suitable  for 
communication,  interpretation  or  processing  by  humans  or  automatic  means.  [DIS] 

Database.  A  collection  of  data  organized  according  to  a  schema  to  serve  one  or  more 
applications.  [DIS] 

Data  Certification.  The  determination  that  data  have  been  verified  and  validated.  (1)  Data 
producer  certification  is  the  determination  by  the  data  producer  that  data  have  been  verified 
and  validated  against  documented  standards  of  criteria.  (2)  Data  user  certification  is  the 
determination  by  the  application  sponsor  or  designated  agent  that  data  have  been  verified  and 
validated  as  appropriate  for  the  specific  modeling  and  simulation  (M&S)  usage.  [DIS] 

Data  Logger.  A  device  that  accepts  protocol  data  units  (PDUs)  from  the  network  and  stores 
them  for  later  replay  in  the  same  time  sequence  as  the  PDUs  were  originally  received.  See 
also:  protocol  data  unit  (PDU).  [IEEE  1278.3] 

Data  Validation.  The  documented  assessment  of  data  by  subject  area  experts  and  comparison  to 
known  or  best-estimate  values.  (1)  Data  producer  validation  is  that  documented  assessment 
within  stated  criteria  and  assumptions.  (2)  Data  user  validation  is  that  documented 
assessment  of  data  as  appropriate  for  use  in  an  intended  modeling  and  simulation  (M&S). 

[DIS] 

Data  Verification.  The  use  of  techniques  and  procedures  to  ensure  that  data  meet  specified 
constraints  defined  by  data  standards  and  business  rules.  (1)  Data  producer  verification  is  the 
use  of  techniques  and  procedures  to  ensure  that  data  meet  constraints  defined  by  data 
standards  and  business  rules  derived  from  process  and  data  modeling.  (2)  Data  user 
verification  is  the  use  of  techniques  and  procedures  to  ensure  that  data  meet  user  specified 
constraints  defined  by  data  standards  and  business  rules  derived  from  process  and  data 
modeling  and  that  data  are  transformed  and  formatted  properly.  [DIS] 

Data  Verification,  Validation,  and  Certification.  The  process  of  verifying  the  internal 
consistency  and  correctness  of  data,  validating  that  they  represent  real  world  entities 
appropriate  for  their  intended  purpose  or  an  expected  range  of  purposes,  and  certifying  them 
as  having  a  specified  level  of  quality  or  as  being  appropriate  for  a  specified  use,  type  of  use,  or 
range  of  uses.  The  process  has  two  perspectives:  producer  and  user  process.  See:  data 
validation,  data  verification,  and  data  certification.  [DIS] 

Dead  Reckoning.  See:  remote  entity  approximation. 
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Deaggregate.  See:  disaggregate.  [DIS] 

Distributed  Interactive  Simulation  (DIS).  A  synthetic  environment  within  which  humans  may 
interact  through  simulation(s)  at  multiple  sites  networked  using  compliant  architecture, 
protocols,  standards,  and  databases  (DoDD  5000.59P) 

E 

Electronic  Battlefield.  See:  synthetic  environment.  [DIS] 

Entity.  Any  component  in  a  system  that  requires  explicit  representation  in  a  model.  Entities 
possess  attributes  denoting  specific  properties.  See:  simulation  entity.  [DIS] 

Environment.  (1)  The  texture  or  detail  of  the  domain,  such  as  cities,  farmland,  sea  states,  etc. 

(2)  The  external  objects,  conditions,  and  processes  that  influence  the  behavior  of  a  system 
(such  as  terrain  relief,  weather,  day,  night,  terrain  cultural  features,  etc.)  [DIS] 

Event.  (1)  An  occurrence  that  causes  a  change  of  state  in  a  simulation.  See  also:  conditional 
event;  time-dependent  event.  (2)  The  instant  in  time  at  which  a  change  in  some  variable 
occurs.  [DIS] 

Event-Driven  Simulation.  See:  event-oriented  simulation.  [DIS] 

Event-Oriented  Simulation.  A  simulation  in  which  attention  is  focused  on  the  occurrence  of 
events  and  the  times  at  which  those  events  occur;  for  example,  a  simulation  of  a  digital  circuit 
that  focuses  on  the  time  of  state  transition.  Syn:  event-driven  simulation;  event-sequenced 
simulation.  [DIS] 

Event-Sequenced  Simulation.  See:  event-oriented  simulation.  [DIS] 

Exercise.  (1)  One  or  more  sessions  with  a  common  objective  and  accreditation.  (2)  The  total 
process  of  designing,  assembling,  testing,  conducting,  evaluating,  and  reporting  on  an  activity. 
See:  simulation  exercise.  Syn:  experiment,  demonstration.  [DIS,  IEEE  1278.3] 

F 

Fidelity.  (1)  The  similarity,  both  physical  and  functional,  between  the  simulation  and  that  which 
it  simulates.  (2)  A  measure  of  the  realism  of  a  simulation.  (3)  The  degree  to  which  the 
representation  within  a  simulation  is  similar  to  a  real-world  object,  feature,  or  condition  in  a 
measurable  or  perceivable  manner.  See  also:  model/simulation  validation.  [DIS,  IEEE  1278.1] 

Field.  (1)  A  series  of  contiguous  bits,  treated  as  an  instance  of  a  particular  data  type,  that  may  be 
part  of  a  higher  level  data  structure.  (2)  An  external  operating  area  for  actual  vehicles  or  live 
entities.  See:  field  instrumentation.  [DIS,  IEEE  1278.1] 

G 

Graphical  Model.  A  symbolic  model  whose  properties  are  expressed  in  diagrams.  For  example, 
a  decision  tree  used  to  express  a  complex  procedure.  Contrast  with:  mathematical  model; 
narrative  model;  software  model;  tabular  model.  [DIS] 

Ground  Truth.  The  actual  facts  of  a  situation  without  errors  introduced  by  sensors  or  human 
perception  and  judgment.  [DIS] 
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H 

Human-in-the-Loop  Model.  See:  interactive  model. 

Human-Machine  Simulation.  A  simulation  carried  out  by  both  human  participants  and 
computers,  typically  with  the  human  participants  asked  to  make  decisions  and  a  computer 
performing  processing  based  on  those  decisions.  [DIS] 

I 

Interactive  Model.  A  model  that  requires  human  participation.  Syn:  human-in-the-loop  model. 
[DIS] 

Interoperable.  Two  or  more  simulations  are  distributed  interactive  simulation  (DIS) 
interoperable  for  a  given  exercise  if  they  are  DIS  compliant,  DIS  compatible,  and  their 
performance  characteristics  support  a  fair  fight  to  the  fidelity  required  for  the  exercise. 
Contrast  with:  compatible,  compliant.  [DIS] 

Interoperability.  (1)  The  ability  of  a  set  of  simulation  entities  to  interact  with  an  acceptable 
degree  of  fidelity.  The  acceptability  of  a  model  is  determined  by  the  user  for  the  specific 
purpose  of  the  exercise,  test,  or  analysis.  (2)  The  ability  of  a  set  of  distributed  interactive 
simulation  applications  to  interact  through  the  exchange  of  protocol  data  units.  [DIS] 

L 

Live  Entity.  A  perceptible  object  that  can  appear  in  the  virtual  battlespace  but  is  unaware  and 
nonresponsive  (either  by  intent,  lack  of  capability  or  circumstance)  to  the  actions  of  virtual 
entities.  See  also:  field  instrumentation.  Contrast  with:  live  instrumented  entity.  [DIS] 

Live  Instrumented  Entity.  A  physical  entity  that  is  in  the  real  world  and  can  be  represented  in 
the  distributed  interactive  simulation  (DIS)  virtual  battlespace  which  can  be  manned  or 
unmanned.  The  live  instrumented  entity  has  internal  and/or  external  field  instrumentation  (FI) 
devices/systems  to  record  and  relay  the  entity’s  surroundings,  behavior,  and/or  reaction  to 
events.  If  the  FI  provides  a  two-way  link,  the  events  that  affect  the  live  instrumented  entity 
can  be  occurring  in  the  virtual  battlespace  as  well  as  the  real  world.  See  also:  field 
instrumentation,  live  entity.  [DIS] 

Local  Area  Network  (LAN).  A  class  of  data  network  which  provides  high  data  rate 
interconnection  between  network  nodes  in  close  physical  proximity.  [IEEE  1278.3] 

M 

Measure  of  Performance  (MOP).  Measure  of  how  the  system/individual  performs  its  functions 
in  a  given  environment  (e.g.,  number  of  targets  detected,  reaction  time,  number  of  targets 
nominated,  susceptibility  of  deception,  task  completion  time).  It  is  closely  related  to  inherent 
parameters  (physical  and  structural)  but  measures  attributes  of  system  behavior.  See  also: 
measures  of  effectiveness  (MOE).  [IEE  1278.3] 

Model.  (1)  An  approximation,  representation,  or  idealization  of  selected  aspects  of  the  structure, 
behavior,  operation,  or  other  characteristics  of  a  real-world  process,  concept,  or  system. 
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Note:  Models  may  have  other  models  as  components.  (2)  To  serve  as  a  model  as  in  (1).  (3) 

To  develop  or  use  a  model  as  in  (1).  (4)  A  mathematical  or  otherwise  logical  representation  of 
a  system  or  a  system’s  behavior  over  time.  [DIS] 

Model/Simulation  Accreditation.  The  official  certification  that  a  model  or  simulation  is 
acceptable  for  use  for  a  specific  purpose.  See  also:  distributed  simulation  accreditation. 
Contrast  with:  model/simulation  validation,  model/simulation  verification.  [DoDD  5000.59] 

Model/Simulation  Validation.  The  process  of  determining  the  degree  to  which  a  model  is  an 
accurate  representation  of  the  real  world  from  the  perspective  of  the  intended  use(s)  of  the 
model.  See  also:  distributed  simulation  validation,  fidelity.  Contrast  with:  model  simulation 
accreditation,  model  simulation  verification.  [DoDD  5000.59] 

Model/Simulation  Verification.  The  process  of  determining  that  a  model  implementation 
accurately  represents  the  developer's  conceptual  description  and  specifications.  See  also: 
distributed  simulation  verification.  Contrast  with:  model  simulation  accreditation,  model 
simulation  validation.  [DoDD  5000.59] 

N 

Network  Filter.  A  system  to  selectively  accept  or  reject  data  received  from  the  network.  [DIS] 

Network  Node.  A  specific  network  address.  See:  node.  Contrast  with:  processing  node.  [DIS] 

Node.  A  general  term  denoting  either  a  switching  element  in  a  network  or  a  host  computer 
attached  to  a  network.  See:  processing  node;  network  node.  [IEEE  1278.1,  IEEE  1278.2] 

o 

Operational  Environment.  A  composite  of  the  conditions,  circumstances,  and  influences  which 
affect  the  employment  of  military  (or  other)  forces  and  the  decisions  of  the  unit  commander  or 
person  in  charge.  [DIS] 

P 

Platform.  A  generic  term  used  to  describe  a  level  of  representation  equating  to  vehicles,  aircraft, 
missiles,  ships,  fixed  sites,  etc.,  in  the  hierarchy  of  representation  possibilities.  Other 
representation  levels  include  units  (made  up  of  platforms)  and  components  or  modules  (which 
make  up  platforms.)  [DIS] 

Protocol  Data  Unit  (PDU).  A  distributed  interactive  simulation  (DIS)  data  message  that  is 
passed  on  a  network  between  simulation  applications  according  to  a  defined  protocol.  [IEEE 
1278.1] 
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R 

Real  Time.  In  modeling  and  simulation,  simulated  time  advances  at  the  same  rate  as  actual  time; 
for  example,  running  the  simulation  for  one  second  results  in  the  model  advancing  time  by  one 
second.  Contrast  with:  fast  time,  slow  time.  [DIS] 

Resolution.  (1)  The  degree  to  which  near  equal  results  values  can  be  discriminated.  (2)  The 
measure  of  the  ability  to  delineate  picture  detail.  [DIS] 

s 

Scenario.  (1)  Description  of  an  exercise  (initial  conditions).  It  is  part  of  the  session  database 
which  configures  the  units  and  platforms  and  places  them  in  specific  locations  with  specific 
missions.  (2)  An  initial  set  of  conditions  and  time  line  of  significant  events  imposed  on  trainees 
or  systems  to  achieve  exercise  objectives.  See:  field  exercise.  [DIS,  IEEE  1278.3] 

SIMNET  (Simulator  Networking).  The  prototype  distributed  simulation  upon  which  DIS  was 
based.  [DIS] 

Simulate.  To  represent  a  system  by  a  model  that  behaves  or  operates  like  the  system.  See  also: 
emulate.  [DIS] 

Simulated  Time.  Time  as  represented  within  a  simulation.  Syn:  virtual  time.  See  also:  fast  time; 
real  time;  slow  time.  [DIS] 

Simulation.  (1)  A  model  that  behaves  or  operates  like  a  given  system  when  provided  a  set  of 
controlled  inputs.  Syn:  simulation  model.  See  also:  emulation.  (2)  The  process  of  developing 
or  using  a  model  as  in  (1).  (3)  An  implementation  of  a  special  kind  of  model  that  represents  at 
least  some  key  internal  elements  of  a  system  and  describes  how  those  elements  interact  over 
time.  [DIS] 

Simulation  Environment.  (1)  Consists  of  the  natural  physical  environment  surrounding  the 
simulation  entities  including  land,  oceans,  atmosphere,  near-space,  and  cultural  information. 
(2)  All  the  conditions,  circumstances,  and  influences  surrounding  and  affecting  simulation 
entities  including  those  stated  in  (1).  [DIS] 

Simulation  Exercise.  An  exercise  that  consists  of  one  or  more  interacting  simulation 
applications.  Simulations  participating  in  the  same  simulation  exercise  share  a  common 
identifying  number  called  the  exercise  identifier.  These  simulations  also  utilize  correlated 
representations  of  the  synthetic  environment  in  which  they  operate.  See:  live  simulation. 

[IEEE  1278.1,  IEEE  1278.2] 

Simulation  Fidelity.  Refers  to  the  degree  of  similarity  between  the  simulated  situation  and  the 
operational  situation.  [IEEE  1278.3] 

Simulation  Time.  (1)  A  simulation’s  internal  representation  of  time.  Simulation  time  may 
accumulate  faster,  slower,  or  at  the  same  pace  as  real  time.  (2)  The  reference  time  (e.g., 
universal  coordinated  time)  within  a  simulation  exercise.  This  time  is  established  ahead  of 
time  by  the  simulation  management  function  and  is  common  to  all  participants  in  a  particular 
exercise.  [DIS,  IEEE  1278.1] 

Simulator.  (1)  A  device,  computer  program,  or  system  that  performs  simulation.  (2)  For 
training,  a  device  which  duplicates  the  essential  features  of  a  task  situation  and  provides  for 
direct  practice.  (3)  For  distributed  interactive  simulation  (DIS),  a  physical  model  or  simulation 
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of  a  weapons  system,  set  of  weapon  systems,  or  piece  of  equipment  which  represents  some 
major  aspects  of  the  equipment’s  operation.  [DIS] 

Site.  (1)  An  actual  physical  location  at  a  specific  geographic  area,  e.g.,  the  Fort  Knox  Close 
Combat  Test  Bed  (CCTB).  (2)  A  node  on  the  network  used  for  distributed  simulation  such  as 
the  Defense  Simulation  Internet  (DSI)  long  haul  network.  (3)  A  level  of  configuration 
authority  within  a  DIS  exercise.  [DIS] 

V 

Validation.  See:  data  validation,  distributed  simulation  validation,  face  validation, 
model/simulation  validation.  [DIS] 

Verification.  See:  data  verification,  distributed  simulation  verification,  model/simulation 
verification 

Verification  and  Validation  (V&V)  Proponent.  The  agency  responsible  for  ensuring  V&V  is 
performed  on  a  specific  model  or  simulation.  [DIS] 

Vignette.  A  self-contained  portion  of  a  scenario.  [DIS] 

Virtual  Battlespace.  The  illusion  resulting  from  simulating  the  actual  battlespace.  [DIS] 

w 

War  Game.  A  simulation  game  in  which  participants  seek  to  achieve  a  specified  military 

objective  given  pre-established  resources  and  constraints;  for  example,  a  simulation  in  which 
participants  make  battlefield  decisions  and  a  computer  determines  the  results  of  those 
decisions.  See  also:  management  game.  Syn:  constructive  simulation;  higher  order  model 
(HOM).  [DIS] 

Wide  Area  Network  (WAN).  A  communications  network  of  devices  which  are  separated  by 
substantial  geographical  distance.  Syn:  long  haul  network.  [IEEE  1278.3] 
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APPENDIX  D  --  List  of  Acronyms 


ADA 

ADS 

ADT 

AFATDS 

AFB 

AFOTEC 

ALQ-131 


AM 

ANIU 

ARIES 

ARPA 

ASAS 

ATACMS 

ATWS 

Bde 

Bn 

C4I 

C4ISR 

CAMPS 

CBS 

CDP 

CEP 

CGS 

Co 

COI 

CPU 

D&SA  BL 

DCT 

DDSA 

DIS 

DMAP 

DoD 

DT&E 

ECCM 

ESPDU 

ETE 

EW 


air  defense  artillery 
advanced  distributed  simulation 
air  data  terminal 

Advanced  Field  Artillery  Tactical  Data  System 
Air  Force  base 

Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  AFB,  New 
Mexico 

a  mature  self-protection  jammer  system;  an  electronic  countermeasures 

system  with  reprogrammable  processor  developed  by  Georgia  Technical 

Research  Institute 

amplitude  modulation 

air  network  interface  unit 

Advanced  Radar  Imaging  Emulation  System 

Advanced  Research  Projects  Agency 

All  Source  Analysis  System 

Army  Tactical  Missile  System 

Advanced  Technology  Work  Station 

brigade 

battalion 

command,  control,  communications,  computers  and  intelligence 
command,  control,  communications,  computers,  intelligence,  surveillance 
and  reconnaissance 

Compartmented  All  Source  Analysis  System  (ASAS)  Message  Processing 
System 

corps  battlefield  simulation 
central  data  processor 
circular  error  probability 
common  ground  station 
company 

critical  operational  issue 

central  processing  unit 

Depth  and  Simultaneous  Attack  Battle  Lab 

digital  communications  terminal 

deputy  director,  system  assessment 

distributed  interactive  simulation 

data  management  and  analysis  plan 

Department  of  Defense 

developmental  test  and  evaluation 

electronic  counter-countermeasures 

entity  state  protocol  data  unit 

End-to-End  Test 

electronic  warfare 


81 


FDC 

fire  direction  center 

FI 

functionality  and  integration 

FTI 

fixed  target  indicator 

GDT 

ground  data  terminal 

GHQ 

general  headquarters 

GNIU 

ground  network  interface  unit 

GPS 

global  positioning  system 

GSM 

ground  station  module 

HF 

high  frequency 

HLA 

high  level  architecture 

HOM 

high  order  model 

HQ 

headquarters 

hrs 

hours 

ID 

infantry  division;  identification 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

JADS 

Joint  Advanced  Distributed  Simulation,  Albuquerque,  New  Mexico 

Janus 

interactive,  computer-based  simulation  of  combat  operations 

Joint  STARS 

Joint  Surveillance  Target  Attack  Radar  System 

JPO 

joint  program  office 

JT&E 

joint  test  and  evaluation 

JTF 

joint  test  force 

km 

kilometers 

LAN 

local  area  network 

LFP 

Live  Fly  Phase 

LGSM 

light  ground  station  module 

LSP 

Linked  Simulators  Phase 

M&IS 

management  and  integration  software 

M&S 

modeling  and  simulation 

MB 

megabyte 

Mbps 

megabits  per  second 

MGSM 

medium  ground  station  module 

MI 

military  intelligence 

mm 

millimeter 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

MOT&E 

multiservice  operational  test  and  evaluation 

ms 

millisecond 

MTI 

moving  target  indicator 

N&E 

network  and  engineering 

NC 

network  coordinator 

NETVisualizer™ 

software  that  displays  real-time  bandwidth  use  in  a  rolling  bar  graph  format 
for  quick  visual  reference 

NIU 

network  interface  unit 

NTP 

network  time  protocol 

O&C 

operations  and  control 
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OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

operational  test  and  evaluation 

OWS 

operator  workstation 

PDU 

protocol  data  unit 

PM 

program  manager 

PME 

primary  mission  equipment 

POC 

point  of  contact 

PIP 

program  test  plan 

PVD 

plan  view  display 

RCL 

radar  components  laboratory 

RPSI 

radar  processor  simulator  and  integrator 

RSR 

radar  service  request 

RWS 

remote  workstation 

SAIC 

Science  Applications  International  Corporation 

SAR 

synthetic  aperture  radar 

SATCOM 

satellite  communications 

SCDL 

surveillance  control  data  link 

SE 

synthetic  environment 

sec 

second 

SGI 

Silicon  Graphics,  Inc. 

SIT 

System  Integration  Test 

SM&C 

system  management  and  control 

SMO 

system  management  office 

Spectrum™ 

an  instrumentation  suite  used  to  measure  bandwidth  utilization 

STARS 

surveillance  target  attack  radar  system 

STRICOM 

U.S.  Army  Simulation,  Training,  and  Instrumentation  Command 

SUT 

system  under  test 

SWA 

Southwest  Asia 

T&E 

test  and  evaluation 

T-l 

digital  carrier  used  to  transmit  a  formatted  digital  signal  at  1.544  megabits 
per  second 

TAC 

target  analysis  cell 

TAFSM 

Tactical  Army  Fire  Support  Model 

TCAC 

Test  Control  and  Analysis  Center,  Albuquerque,  New  Mexico 

TEXCOM 

U.S.  Army  Test  and  Experimentation  Command 

TRAC 

U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

UAV 

unmanned  aerial  vehicle 

UDP 

user  datagram  protocol 

UHF 

ultra  high  frequency 

V&V 

verification  and  validation 

VDP 

VSTARS  data  packet 

VHF 

very  high  frequency 

VSTARS 

Virtual  Surveillance  Target  Attack  Radar  System 

VV&A 

verification,  validation,  and  accreditation 
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w&c 

WAN 

WSMR 

XPATCHES 


verification,  validation  and  certification 

wide  area  network 

White  Sands  Missile  Range 

E-8C  synthetic  aperture  radar  simulation  developed  by  Wright  Laboratory, 
Dayton,  Ohio,  and  Loral  Defense  Systems,  Goodyear,  Arizona 
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